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INTRODUCTION 


The Subject 





_The controversy surrounding the procurement of the C-5A Galaxy, 
a new transport aircraft obtained for the U.S. Air Force, has become 
a cause celebre for Congressional and Public critics of Department of 


I iad e e 
Defense spending. The occurrence of exceptional cost over-runs in 





this program has been cited as evidence of poor management practice. 
One cannot refute this contention, for great excesses in spending 
above budgeted Nevels per force demonstrate poor management. However, 
it is important to determine which management techniques have failed, 
‘if circumstances such as those experienced in the Galaxy program are 
le be avoided in the future. . | 
There is little doubt that the Galaxy program stretched the 
state-of-the-art, to a degree not originally anticipated, in aero- 
nautical engineering and manufacturing, when it sought to reach new 
technological plateaus. It is now evident that economic, as well as 
technological factors, contributed to unforseen developments. The 
initial planning estimates, made in 1964 and based on a preceding 
five-year perodiarked by a relatively stable economic environment, 
would certainly have been revised upward if conditions experienced 


during the succeeding five years had been anticipated. 
Though the Galaxy program did experience certain technolog- | 


ical and economic consequences which might be regarded as extraordinary, 








it has not been unique in terms of unanticipated expenditures. It 

has been observed that "initial cost estimates of major weapons sys- 

tems have historically escalated, just as they have in the eon 
One should not conclude from the preceding that problems in 


controlling cost increases are peculiar to government. Much of the 


unexpected expense for the Galaxy had to be absorbed by Lockheed Air- 
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craft Corporation, the prime contractor, since initial contracts, based 
on Lockheed estimates, did not provide for the expenditures that were 
required. Corporations other than Lockheed have also experienced simi- 
lar predicaments. For example, General Dynamics, while still trying to | 
recover from the financial calamities experienced with the F-111 mili- 
tary aircraft and the Convair 880-990 commercial jets, is now encoun- 
tering losses in shipbuilding programs; all due to unplanned cost. 

It is possible to gain some insight into the difficulties that 
havé been described if recognition is given to the commonality which is | 
displayed: each of the ventures which has been cited relied heavily on 
research and development to provide techniques and components needed 
for the achievement of program objectives. This fact would then lead 
one to conclude that a major contributor to problems of this type is a 
growing cost of technological advance. 

Improvements in technology derive Sheol: from research and 


development efforts. It has been observed, in this regard, that as 


° ee ee ee ee = 


technology does advance, the scientific disciplines, to be considered 


aed Forces Manegement, (July, 1969), p. 58. = \-— 


2"General Dynamics: -In Trouble Again," Business Week, (October — 
4, 1969), pp. 48-52. 
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in seeking continued advance, become more esoteric in nature and great- 


x 


er in number. Thus, as program objectives move to successively higher 


levels of peices ticne the number of alternatives available for 
realization of the objectives increases, as does the costs of the 
alternatives. 

‘The problem, then, is one which has increased in significance 
in the face of technological advances, and derives from difficulties 
encountered in both selecting appropriate courses of action for a- 


mone objectives and in properly forecasting and controlling the 


expense to be realized in pursuing the selected course. It has been 
recognized that there is a growing need for improved techniques to be 


made available to managers for the purpose of making hard choices in 


the funding of research and development efforts. 


The Objective 
A major facet of the research and development project man- 
lager's mole is essentially one of control. This is exercised by 
monitoring project progress and ensuring that achievement and asso- 
ciated expenditures are kept within planned limits. This function 
Will also include the task of affecting certain modifications, as 
mecessary, tO correct deviations from the plan.7 | 
IMarvin J. Cetron, "Technological Forecasting: A Prescrip- 


tion for the Military R & D Manager," Naval War College Review, 
XXI (April, 1969), p. 14. 


Sete E. Souder, "Experiences with an R & D Project Con- 
trol Model,'' IEEE Transactions on Engineering Management, EM-15 


(Harehy 1968), p. 40. 
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Control of research and development is, of course, closely re- 


lated to the planning function which entails the selection of pro- 
grams and the scheduling and allocation of funds. Control and plan- 


ning interact principly when information, obtained through control, 


causes changes in program scheduling or fund allocation. For instance, 


the progress realized in a program may result in that program being 
terminated entirely, or being expanded at the expense of some other 
program. 

Difficulties are encountered in research and development 
Management which are due to an intrinsic uncertainty. prectenaily ant 
is impossible to discern whether a program is on the verge of failure 
or success. This uncertainty is also quite prevelant in program se- 
lection since @ prtorz estimates are often little more than educated 
guesses, and the selection of programs on the basis of merit is often 
difficult. 

The objective of this paper is to develop a technique for 
evaluating the relative value of proposed programs, so as to assist 
in program selection; for monitoring program progress and evaluating 
observed deviations; and for providing information to be used in ad- 
justment of original planned program content and expenditure levels. 
Proposed programs are to be assessed on the basis of some selected 
measure of utility so that available funds may be allocated among 
alternative proposals in order to realize a maximum retum. Program 
progress is to be measured in terms of achievement versus cost and 
achievement versus time. Thus, it will be possible to identify pro- 


grams that either fail to meet prescribed time schedules or exceed 
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budgeted expenditures. Finally, using data obtained from the analysis 


of program progress, it will be possible to recommend changes in pro- 
gram mix in keeping with observed results. 

As was stated, it is intended that the technique, which is to 
be developed, will be useful only in selecting from among proposed 
programs. It is not intended to formulate proposals. The actual 
formulation of proposals must be done on the basis of objectives and 
applicable policy considerations. This requires a certain amount of 
subjective evaluation. Only after this evaluation Neeiccn completed, 
and acceptable candidate programs selected, is it possible to conduct. 

jan objective analysis on the basis of the parameters mentioned above. 

m ardor touftacititate the Hevelornent of an analytic technique, 
jit will be necessary to make certain assumptions regarding the struc- 
ture of subject programs, the statistical independence of programs 
| and" the probabilistic character of research and development efforts. 
Petever, necessary assumptions will be examined to ascertain the 
 —_ of any inaccuracies which may be introduced. 7 

The technique which is to be generated will derive from man- | 
agement systems common to Department of Defense research laboratories. 
As a result, it will not be universally applicable in all management 
Situations. However, since the intent is to simply demonstrate a ~ 
means of incorporating certain analytic fechous inte the management 
process, the methods themselves should be easily adaptable to other 


Management systems. Throughout the ensuing analysis, appropriate 


analogy will be made, as required, to assist in relating the dis- 


cussion to non-defense oriented circumstances. 








Definitions : 





This paper constitutes a merger of several disciplines which 
are of different nature and utilize a variety of nomenclatures. Since 
the discussion is foremost intended as a business-oriented work it is 
assumed that the reader is familiar with the terminology associated 


with management science and economics. Moreover, due to the essential 
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quantitative character of the paper, it must be presumed that there is 


also some familiarity with the semantic peculiarities of mathematics; 
particularily probability theory and statistical analysis. However, 


certain terms which lack precise meaning, and are used frequently in 


~ 


this paper, do deserve definition as -follows: 


} 


the state-of-the-art in any technological discipline. The term research 


i 
is usually associated with those activities concerned with basic con- _ 
i 


1. research and development.--activities which seek to advance > 
| 


cepts and theories and which devote considerable effort to identifying 


and defining topics not previously studied. Development, on the other 


nee 


hand, is generally considered to include those activities directed at 


~ meeememees - 


refining and extending knowledge of existing topics. Research and 


asen- 


development are considered together since research prompts development 


—— 


and since both are endeavors which, presumably, have not previously 


been conducted. 


2. project.--any organized research and development activity, 


conducted by a single group and having defined objectives. A project 


is the lowest level of research and development organization, and can- 


not be decomposed into other activities which are capable of satis- 


ying _project_objectives. uae | 








3. program.--a research and development effort composed of 


several projects. A necessary characteristic of a program is that its 


objectives can be satisfied even if one or more of its constituent 


ane 


parts, the projects, are discontinued. 


The Procedure 
Chapter I, "A Review of Current Methods," presents a survey 
of efforts by other authors in developing analytic procedures for use 
in planning and controlling research and development programs. The 
purpose of this survey is to describe the attempts that have been made 


in the past and to evaluate the degree of success that has been real- 


i 

| 

| ized for the purpose of identifying difficulties which have been en- | 

countered. This survey is also intended to provide a general intro- | 

ction to the elements of quantitative management HENGE 

Chapter II, "Description BE the Technique," examines in depth 

| the two i ae components of quantitative research and development 
management Syctems, project Selection and control. This description 
is in the form of an examination of various techniques that have been 
developed in the past for use in selection and control. This chapter 
forms the foundation for the techniques to be originated in this paper. 

Chapter III, "Construction of the Model," presents the deriva- 

tion of the selection and control techniques which are the subject of 
the paper. The derivation demonstrates a method of organizing elements 


of the procedures discussed in Chapter II such that an improved ap- 


proach to the problem of project selection is obtained. Also, a : 


technique for optimizing project selection is presented. The chapter 











concludes with the development of a method for controlling research 
and development projects that is based on statistical inference. : 
Chapter IV, "Conclusions," summarizes the status of quantita- 


tive selection and control techniques as it stands at present, and 


Chapter III. In addition, elements of the subject techniques which 


| 
reviews the role that is anticipated for the techniques offered in 
deserve further refinement are discussed. 


Appended to the text, in Appendix A, is a sample of mat- 
| 
erials which are considered useful in the implementation of quanti- 


| 
le 
tative selection methods. Appendix B presents a tabulation of the 


i 
| 

‘cumulative Poisson distribution which is used in the control technique 

| 


of Chapter III. Finally, Appendix C presents a partial tabulation of 


are ew meee ne ae + 


s the chi-square distribution, also referred to in Chapter WAI 








CHAPTER I 





A REVIEW OF CURRENT METHODS 


Focusing On the Problem 

As described in the introduction, the problem under considera- 
tion is one of developing more effective methods for the management of 
research and development programs. A specific provision was included 
which precludes processes involved in the formulation of programs. 
These processes, which are generally classified under the heading of 
normative forecasting, have to do with the identification of goals and 
selection of candidate programs on the basis of policy criteria; seep 
ly excluding considerations of economy and technological feasibility. 

After the normative forecasting process is complete, and a set 
of acceptable research programs has been generated, management then 
tries to determine which programs from the set to pursue (project se- 
lection), how much money to spend on each program (budget allocation), 
and the size of the total budget for all the programs selected (budget 
determination).! The first two of these functions,: project selection 


and budget allocation, constitute the key areas of interest for this 


paper. a | 
As was stated earlier, project selection is essentially a i 
le, M. Rosen and W. E. Souder, ''A Method for Allocating R & D 


Expenditures ,'' IEEE Transactions On Engineering Management, EM-12 | 
(September, 1965), p. 90. | 





—) 





10 





planning activity. However, budget allocation is a planning problem | 


also; it takes place at the start of a program or at a review point 
where some change of plans is indicated. The key decisions in budget 


(or resource) allocation involve the rates of resource expenditure and 


— ee oe 


the planned dates for the commencement of an activity. ! 
‘It should be re-iterated that, since attention is going to fo- 


cus on planning and plan review, consideration is appropriately due 


Sees Be ee a see 


the associated control processes. In this respect, examination of 
Management processes will include those concerned with monitoring pro- 


gram progress and ensuring that prescribed limits are observed. 


The Analytic, Framework 


In the abstract, one can consider research as simply the pur- 
ichase of information upon which to base later and better decisions. All 
but one of these decisions is whether or not to continue purchasing in- 


‘formation (that is research). The one exception is the final decision, 


l 
i 
| 
| 
i 
{ 
| 
(which is whether or not to commercialize the results. The decisions to 
continue a research effort, and the subsequent decisions concerned with 
how to conduct the research do appear to be sequential in nature. This 
sequential decision making process has been the subject of descriptive 
work in the past.” It is worthwhile to note that the applicability of 
IR. S. Rosenbloom, "Notes on the Development of Network Models 


for Resource Allocation in R & D Projects,'' IEEE Transactions on Engi- 
neering Management, EM-11 (June, 1964), p. 62. 


2p, L. Marples, "The Decisions of Engineering Design," IRE 


Transactions on Engineering Management, EM-8 (June, 1961), pp. 55-71. 
and T. A. Marshak, "Strategy and Organization in a System Development 
Project,'’ The Rate and Direction of Inventive Activity (Princeton, New |. 
Jersey: Princeton University Press, 1962), pp. 461-475. 











| 
| 


the sequential decision model for research efforts is a function of 


the degree of structure that is evidenced in the effort. For example, 
in basic research, which is fairly unstructured, the model should be 
quite appropriate. In developmental design, however, decisions to 
commence a phase of the program may be made before preceding phases 
are Popiete. In this case, then, some inaccuracies may be intro- 
duced. Nonetheless, for the pruposes of the present discussion, the 
model can be presumed adequate. 

The concept of sequential decisions is only one ingredient in 
a characterization of research and development processes. Previous 
studies, for instance, have identified certain parameters as being 


significant in research efforts. Among these are the magnitude of 


‘!total expenditures, duration, work scope, activity content, geographic 


dispersion of related elements, and others. -Furthermore, any one or 
more of these parameters may be selected as a primary measure; de- 
pending on the type and objectives of the analysis for which the 
characterization is intended. Of the numerous parameters which might 
be used, certain ones have been found to be more useful as constituents 
of a model for the research process. These are: 

1) the items worked on, 

2) the activities (operations) performed on these items, 

3) the duration of each assignment, 

4) the effort (or resources) utilized by each assignment. 


Furthermore, the dynamics of the research effort have been found to be 


adequately represented by the following: 


1) the start and stop time of each assignment, in calendar 
time, : 
2) the total effort content of each assignment. 


fattest ns a Der ctibie= is vere es a wea ree’ 
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| 
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3) the distribution of this effort content over the duration 
of the assignment. : 


A final aspect of the research process which should be con- 


sidered in an analytic characterization is the representation of the 


economies of operation. Here, economy of operation relates to the rate 
at which resources can be effectively employed. In this respect, the 
research process is partitioned into three funding Stages. The first 
stage lies below a "critical" cost level, such that any allocation be- 


low this level yields no return. Above the critical cost level is a 


i 
| 
istage, within which, progress is a monotonically increasing function 


of cost. This mid-stage, which can be called the "adequate" stage, 


| 
i 
! 
| 
j 


;has an upper termination at a funding level called the "satiation" 
{ ° e e e 
;cost. Above this cost level increases in cost have no corresponding 


ieee: in progress. 

| In concluding this discussion of the analytic approach to re- 
search and development planning and control, it is considered that the 
linteractions of the Bereta components of the management system can be 
summarized by graphic display as shown in Figure 1. It is intended 
that this should illustrate the inter-related nature of planning and 


control, and indicate the fashion in which each associates with pro- 


gram operation. 


Ip. V. Norden, "On the Anatomy of Development Projects," IRE 


Transaction on Engineering Management, EM-7 (March, 1960), p. 37. 

aoe J. Freeman, "A Stochastic Model for Determining the Size 
and Allocation of the Research Budget,'' IRE Transactions on Engineering 
Management, EM-7 (March, 1960), p. 4. 
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~ Some Current Techniques 


Numerous authors have offered quantitative techniques for the 


management of research and development. ! Of these, the techniques ! 
that have objectives similar to those of this paper are described below: 
.1) Asher, 1962.--This method was derived for the purpose of : 
| 
man-power allocation in a pharmaceutical firm. The method considers a : 
number of alternative projects on the basis of discounted net value, if 
successful; probability of success; and man-hours (resources) more 
including skill levels. From these data an expected discounted value 
is computed. Next, linear programming is used to assign various re- 
Mearen teams, characterized by skill-levels represented, to maximize 
the expected return subject to a constraining number of hours available 
for each team. 
2) Atkinson and Bobis, 1969.--This is a method for determin- 


ing money to be spent on product oriented research programs. The 


ne 
2 
° 
= oe ont ee 


method commences by estimating the density function of the probability 
jof a project succeeding for a given expenditure, This distribution 
is given in the form of a logistic function. On the basis of this 


function, the probability of completing a program within a given time, 


es 


for a given expenditure, is determined. 


leor a brief but thorough review of current techniques, in- 
cluding an exceptional bibliography of the topic see M. J. Cetron, J. 
Martino and L. Roepcke, ''The Selection of R & D Program Content--Survey 
of Quantitative Methods,'' IEEE Transactions on Engineering Management, 
EM-14 (March, 1967), pp. 4-15. | 


2p, T. Asher, ''A Linear Programming Model for the Allocation of 
R and D Efforts,'' IRE Transactions on Engineering Management, EM-9 


(December, 1962), pp. 154-57. | 








The next step in the analysis involves a determination of com- 


a 


mercial value as a function of completion time. This, in conjunction 


with estimates of completion time yields an expected commercial value. 





Having these data, it is then possible to optimize the expected value 
of research on the basis of annual expenditures for the alternative 
| programs being considered. Due to the formidable nature of this op- 


Weimization problem which results from the numerous variables involved, 
/an iterative optimization technique is employed. + 


r) 
4 


3) Dean and Hauser, 1967.--The employment of this method com- 


imences with a decomposition of the research program into the following 


~ 


' catagories: 


a) Qualitative Materiel Development Objectives (QMDO)--the 


. several major goals of the research (effort); corresponding approximate- 


ly to program objectives as defined in the Introduction. 


b) Materiel Concepts--the various different component efforts 


ete oo ewe ee rs te 


(of the QMDO (for instance, if a QMDO is the development of a missile, 
| these might be guidance, propulsion and structure); corresponding 
approximately to program content. 
¢) Technical Approaches--the several alternative methods of 
achieving a materiel concept; corresponding to different projects 
which satisfy the same objectives. 
The utility analysis of the research program is affected by 


first estimating the probability of success and cost for the technical 





a C. Atkinson and A. H. Bobis, "A Mathematical Basis for the 


Selection of Research Projects,'' IEEE Transactions on Engineering 
Management, EM-16 (February, 1969), pp. 2-8. 


| 
| 











approaches. With these data it is then possible to determine the al- 


location of funds, among the technical approaches for a given material 
concept, which maximizes the probability of the material concept suc- 
ceeding; subject to the constraint of a fixed amount being available 
for the material concept funding. 

Rye suming. that, in order for a QMDO to succeed, all component 
‘materiel concepts must succeed, it is then possible to find the fund- 


‘ing levels for materiel concepts in order to maximize the QMDO proba- 


2 eames wee wes ware 


bility of success; subject to a budget constraint. 


This particular method has been found amenable to optimization 
iby dynamic programming. The authors maintain that such a solution 
! 
contributes flexibility and speed to solution, and is particularily 
cuitabte BOemmnCuLyYPpemoLt structure used in decomposition of the 
! 
| program. ! 
| 4) Hess, 1962.--Hess introduced the first dynamic programming 
‘solution to the allocation problem. His solution assumes values for 
certain parameters, among them being the probability of technical 
meuccess, expected profit as a function of time of completion, and an 
assumed measure of usefulness deriving from prior research. Using the 
dynamic technique that is developed, it is possible to obtain an op- 


timal allocation of funds among programs and budgeted rates of ex- 


penditure for each program; subject to appropriate overall budget 


1B. V. Dean and L. E. Hauser, "Advanced Materiel Systems Plan- 
ning," IEEE Transactions on Engineering Management, EM-14 (March, 1967), 
pp. 21-43. . 
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constraints. 


Aspects of Uncertainty 


Complications attendent to uncertainty arising from opera- 
tional situations are by no means limited to management of research 
and development activities. Similar problems are common to the 
ee eement of most business efforts. This is evidenced by the de- 
| velopment of disciplines such as statistical decision theory, and 
I their rise to prominence in management science. Research efforts, 
eer, are somewhat unique in that they demonstrate a degree of 
Pe nous uncertainty not characteristic of other management pro- 
Pe... Even if environmental factors (such as market conditions and 


t 
the supply of materials, labor and funds) were deterministic, the re- 





sults of research and development programs would not be. Thus, the 


— 


Significance of uncertainty in research management has added impor- 





| tance. 

! As is indicated by the preceding description of some available 
techniques, this uncertainty is usually accounted for by means of a 
“probability of success" term. This approach to the problem, however, 
has some disadvantages. 

To begin with, there is no universally pecontaa meaning of 
probability of success. Most authors account for this, to some extent, 
by allowing for a probability of technical success and, also, for 

Is. W. Hess, "A Dynamic Programming Approach to R and D Budg- 


eting and Project Selection,'' IRE Transactions on Engineering Manage- 
ment, EM-9 (December, 1962), pp. 170-79. 
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commercial success. Even if one were to limit consideration to 


technical success alone, a probability measure is inadequate, since it 
does not allow for varying degrees of success which might result. In 
reference to the probability of success concept, Baker and Pound ob- 
served that “at best, it is extremely difficult to understand what is 
meant by such a term. "1 7 
The use of probability of success is further Porm uicated by 
the fact that it is extremely difficult to estimate a value for such a 
parameter. In the work done by Asher, for instance, probabilities 
‘xanged between five and twenty-five chances of success per ten 
| thousand. 2 Needless to say, even the slightest variation in estimates, 


lin cases such as this, would have disproportionate effects due to the 


small order of magnitude involved. There are two principle methods 


tne ome met 


of estimating probability of success. First is by obtaining the sub- 


ae oe eee a 


jective judgements of persons experienced in the subject area. Second 
is by analysis of data obtained from similar prior efforts. The main 
shortcoming of the first method lies in the bias that is likely to be 
introduced. The second is found disadvantageous since probability of 


success is not a fixed quantity, but is a function of the state-of- 
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the-art and level of knowledge prevailing at the time of the estimate. 


In. R. Baker and W. H. Pound, "RED Provectscelection: Where 
We Stand,"' IEEE Transactions on Engineering Management, EM-11 (December, 
m964), p. 129. j 


2asher, "A Linear Programming Model for the Allocation of R & D 
efforts,’ p. 155. 


SRosen and Souder, "A Method for Allocating R Gal Expenditures,” 
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A final complication realized in using probability of success 


estimates is that there is no reason to assume that the probability of 
success and level of funding are independent. On the contrary, the 
converse is almost surely true. Despite this, only the method of Hess 


gives consideration to the apparent dependency. 


Data Requirements 


Since the descriptions of methods given earlier were somewhat 
abbreviated, the data requirements appear deceptively minor. In gen- 
feral all techniques, developed to date, require information of past 
efforts. As such, many of the methods are not applicable to organiza- 
tions that are initiating research and development programs. In some 
-!cases, such as that of Hess, the data requirements and assumptions are 
quite restrictive. As Hess points out, a considerable amount of work 
would be required before his method could be applied.! 


Tne fact of the matter is that, at the present time, there is 


research and development selection decisions. Even if a great deal of 
past data were available, it is not clear how this might be useful in 
evaluating current, different projects; for reasons which have already 
been discussed. 

As a result of this paucity of "good'"' information, compensatory 
methods have been undertaken. As was mentioned in the discussion of 


i 
| 
1 
i 
| 
| 
sufficient data available upon which to base formally structured 
na one approach has been the use of subjective estimates, an 


lpaker and Pound, "R & D Project Selection: Where We Stand," 
p. 128. 
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approach requiring judgmental values for costs, returns, probabilities 
of success, and so forth. The difficulty with these methods has been 
that there is generally a prevalent bias which may render the data 


unreliable. 


For instance, persons wishing to 'look good" will con- 
sciously or unconsciously underestimate the potential of proposed 

projects. On the other hand, there are the incurable optimists who 
jthink they can produce results worth millions, at very low cost, and 


2 


in a short period of time. If anything, the most common tendency 


is towards over optimism. This, of course, is the error that is most 


Optimization Criteria 


Pomtlestcrmea project selection implies, efforts of this sort 


| 
costly.> | 


iseek to evaluate and compare alternative proposals for the purpose 
‘of selecting those identified as holding the most promise. It is obvi-° 
ous that this process requires a basis for selection. The most common 


‘criteria that are used are some form of profit or return on investment; 











either can be, and are, computed in sundry fashions. This limited view 
of what constitutes an adequate criterion has undoubtedly grown out of 
the tendency of business, in general, to use profit or return on in- 
estment. However, as Dean and Hauser have noted, el a method is 


formulated for application in a particular situation, the concern 


i 
libid., p. 125. 


2R. J. Freeman, "A Stochastic Model for Determining the Size 
and Allocation of the Research Budget," p. 2. . | 


3R. M. Anderson, "Handling Risk in Defense Contracting," Harvard 
Business Review, XLVII (July-August, 1969), p. 92. — i: 








should not be with the development of criteria, but with the formula- 


tion of models which are adaptable to criteria. The more general in- 
tent should be to generate methods which enable the manager to syn- 

thesize the information that is available, so that alternative cri- | 

teria can be eee : 

Progress to Date | 

Several studies have been conducted for the purpose of deter- 


a’ the extent to which quantitative techniques have been incorpor- 


ated into program selection and budgeting efforts.” All of these 





have indicated that use of these techniques has been slight; despite 

i the parrot trend of capitalizing on etch methods in other manage- 
_Iment efforts. 

_ One survey Perciaded that a major reason for the lack of signi- 


'ficant employment has been that the methods have not been thoroughly 


e 
ee 


tested, using real data.” A more recent examination, completed during 





‘the year prior to this writing, observed that a more likely reason is’ 
that the rationale for the selections is not subject to external vali- 


4 . 
dation of any kind. In most cases, it was concluded, the only test of 
| 
Inean and Hauser, "Advanced Materiel Systems Planning," p. 22. 


“alleen and Pound, "R and D Project Selection: Where We Stand," 
and Cetron, Martino and Reopke, "The Selection of R & D Program Content- 


-Survey of Quantitative Methods." | 


3paker and Pound, "R and D Project Selection: Where We Stand," 
p. 130. | 

4Robert Ayres, Technological Forecasting and Long-Range Plan- - - 
ning (New York: McGraw-Hill Book Company, 1969), p. 198. 
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validity is the internal consistency and reasonableness of the com- 
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parison procedures. Such a criterion, however, is usually only fully 
satisfactory to the model designer. 


One should not be too hasty, though, in discounting the 
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utility of quantitative techniques simply because ''they do not prove 


out", This, frequently, is a characteristic of any statistical de- 


cision-making technique. 


Two conclusions might be made at this point. The first is 
| that the problem at hand, that of project selection, and so on, may be 
| deserving of some attention to developing methods of solution which are 
more practical in nature, vice those which are theoretically more 
mae Se- Second is a need to recognize that quantitative techniques 
| should not be expected to replace the decision-maker. The impact of 
| 


these techniques is not in problem solving, but in problem formulation: 


the way in which managers conceptualize their problems. 











‘ 
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DESCRIPTION OF THE TECHNIQUE 


. The techniques to be discussed in this chapter are quantita- 
tive techniques similar to those examined in the preceding chapter. 
Before continuing, ASsarine it should be recognized that techniques 
such as these are not used to any great extent in existing research 
and development management systems. Most systems, today, still rely 
on the basic methods of more conventional systems; being oriented 
towards control on the basis of budgetary variances. 

The management of research and development solely on the 
basis of financial considerations, without a sincere attempt to in- 
corporate progress reporting, in some instances degenerates into a 
contest among program managers, which tests political rather than 
technical skills. Several rather familiar methods of program manage- 
ment have been spawned under these conditions. One of these is the 
squeaking wheel method of allocating resources. Using this method, a 
manager, preparing for the allocation of funds, will reduce resources 
for all projects and then wait to see which project. manager expresses 
the loudest complaint. On the basis of the most insistent squeaking 
then, resources will be peerores until budgetary levels are reached. 


A second popular management technique is that which seeks to 


perpetuate the Glorious Past. This method dictates that a group within 


‘CHAPTER II 
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the organization, which has recently demonstrated notable success, 


should gain acceptance of research proposals for the next year, or | 
| 
five years, or for some other appropriate length of time. This doc- 
trine, which is founded on the premise of "once successful, always , 
successful" permits the pursuit of research, independent of usefulness 
or efficiency. | e | 
A final management approach to resource allocation is based on 
the white charger technique. To implement this method, management calls 
on various departments or groups within the organization to hold forth 
with well-rehearsed presentations complete with multi-color graphs and 


handouts. That group which provides -the most impressive forensic dis- 


| play is then rewarded with increased resources. 
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Though techniques such as these seem to be tragi-comic in the 


context of management science, their popularity and frequency of use 


ip a 0 tare 


is pieseantial.- The fact that methods such as these are used to the 


extent indicated serves to illustrate the need for objective consider- 


ation of progress and achievement in research and development. 
The process of program selection, regardless of the type of 
quantitative selection technique that is employed, must include cer- 
tain operations and procedures. These necessary Seen common to all 
selection processes, constitute the subject material of this chapter. 
After these steps are examined and analyzed, the following chapter will | 


describe the methods to be utilized. 


In J. Cetron, "Technological Forecasting: A Prescription for: 
the Military R & D Manager," Naval War College Review, XXI (April, 1969) 


Dm. 29. 


en ee 








Project Selection 


The process of selecting research and development projects is 


closely akin to most other decision-making processes in that it re- 
quires the evaluation of alternatives preparatory to actual selection. 
In this respect, then, project evaluation implies a prtor?t technical 
evaluation to determine the degree to which projects contribute to the 
| accomplishment of established goals and objectives. This is in con- 
trast to a conventional meaning of evaluation, which connotes a post- 

i mortem to analyze the effectiveness and efficiency of the conduct of 

an accomplished project. Project evaluation is, therefore, intended to 
Piieinate some sent of a Bferacteristic quantity or expression which 


Bee sents a project and will enable a value comparison with other 
| 


projects competing for funds and resources. 


Objectives 
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Prior to the commencement of project evaluation, it is 
necessary to identify some broad objectives of the entire selection 
process. These objectives may simply be in a statement of guiding 
policies, or they may be more closely defined. 

Research and development objectives are usually products of 
long-range planning. This planning process commences, as does plan- 
ning for non-research and development activities, with consideration 


of existing plans and policies, the state-of-the-art in disciplines 


of interest and the competitive environment. The results of this 











portion of the planning process can be considered to be "strategic 


e e 1 e e e e 
objectives" since they derive from consideration of current re- 


— 


search and development conditions and focus on projected conditions 
and competitive position. 

The nature of strategic objectives can be illustrated if one 
considers a hypothetical case: Cunisibe Metals, mes is a medium-size 
materials research organization which limits its efforts to non-ferrous 
applications. In the past, Cunisibe has usually expended about 75 per 


| 
| 
‘cent of its basic research budget in the copper-bronze field. However, 
'Cunisibe's position in this field has never improved substantially in 


i 
i 
j 


ha 


the face of considerable competition from the research divisions of 
the nation's three largest copper-producing companies. 

: Cunisibe's most recent planning effort has high-lighted two 
‘particular environmental factors: a | 
1) The questionable long-range position of copper-bronze | 


‘materials due to recent shortages of domestic raw materials and an 


eat es 


even more questionable stability among African suppliers. 
2) A growing interest in ocean-engineering and efforts to 
exploit ocean-resources. | 
As a result of these factors, then, Cunisibe formulated a 


strategic objective aimed at gaining a national position based on an 


é 


the terminology used in this section regards strategic, man- 
agement and operational objectives, will correlate with the terminology 
,of R. N. Anthony in Planning and Control Systems -~ A Framework for 
Analysis (Boston: Harvard University, 1965). | 


2 e e e * e e e ; e ; 
This is a fictitious organization. However, the circumstances — 
described resemble those of an actual research organization. 
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expertise in developing materials appropriate for use in the ocean 


environment. This strategic objective will cause Cunisibe to shift 
emphasis from copper-bronze research to endeavors in utilizing such 
materials as tungsten and magesium. Likewise, it will open new dis- 
ciplines, such as galvanic-corrosion control. . 

| Strategic objectives, however, cannot afford adequate guidance 
for immediate applications. They simply identify long-term needs. 


The next step, then, towards establishing operational requirements is 


— 


to determine the nature of corporate deficiencies regards the es- 
tablished strategic objectives. This portion of the planning process 
is essentially a deficiency apalvaiia OF current capabilities and re- 
sources. The result of this analysis will be "management objectives" 
| which will be applicable in short-term planning. These management 
| objectives will identify subject areas deserving attention in organ- 
izing programs. Following the previous example, management objectives 

iat Cunisibe for the first few years might include: 

1) Acquisition of personnel with expertise in the fields of 
Tungsten alloying processes, scintered metallurgy and metal-fibre 
| growth techniques. 

2) Development of research facilities adequate to support 
new areas of interest. | 


An examination of management objectives would reveal that, 


though they are more precise than stategic objectives, they are too 


general to be immediately useful. Thus, it is necessary to translate 


Management objectives into "operational objectives" which can be 
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directly related to particular research endeavors. These pperational 
objectives are the criteria that are used in the comparative analysis 
of research proposals. 

Operational objectives are usually couched in terms of certain 
technologies or topic areas which have been identified as being of 
interest. However, for production-oriented, commercial firms oper- 

iational objectives may be related in more economic terms. This, nat- 


lurally, results from the fact that such firms have objectives which, 


jgenerally, are more readily expressed in terms of economic indicators. 
ov example, one market-oriented firm adopted the following as the ; 
{ 


| 


pt 
objectives of research and development programs: . 


iy Foemaximize expected profit. 
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{ 
| 
2) To maximize expected research successes. 
3) To achieve a return on expenditure (ratio of maximum ex- 


pected profit to total expended) of at least 55.0. 


ones ee cemetnre sowens 


Regardless of whether a research and development organization 
is profit oriented or non-profit, or devoted to product development or 
process Pe carch. the exercise of formulating objectives is essential 
to any attempts at selecting program content on the basis of struc- 
tured, quantitative procedures. As will be Seats in following 
sections, these objectives are needed if candidate programs are to 
be compared in an objective fashion, and if ultimate program progress 


is to be monitored meaningfully. 


Jposen and Souder, "A Method of Allocating R & D Expenditures," 








Utility Analysis 








A significant characteristic of the process of formulating 
objectives, that has been described in the preceding, is es it is 
almost exclusively a qualitative exercise. Consequently, the de- 
cisions, made in arriving at BR ee ties: have little utility in terms 
of a quantitative selection technique. This section will be devoted 
to describing some traditional operations research procedures which 
can be used to relate quantitative techniques to qualitative ob- 


meectives. 
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In the preceding example the formulation of objectives progres- 


sed to the point of operational objectives, which are to provide the 

necessary decision rules for program selection. The next step in the 

analysis must then cast program alternatives in terms which are 

amenable to manipulation, in order that decision males may be applied. 
The quantitative characterization of alternatives is most 


easily done using "figures of merit" or "value functions". Either 


of these terms refers to the assignment of a number or expression, to 


each alternative, which represents the utility or desireability of 
the Sse: The measurement of utility in this fashion is an 
essential menedient of traditional operations pesearch methods such 
as game theory, allocation problems and solutions of competitive 
situations.! Though a variety of utility measurement methods are 


available, an examination of two of them will adequately illustrate 


IR, L. Ackoff and M. W. Sasieni, Fundamentals of Operations 
Research (New York: John Wiley and Sons, 1968), pp. 49-56. 
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the principles involved. 


Utility analysis, seeks to determine the value or value function 


to be attached to each research proposal. Value, here, should be 
thought of as expressing the "net worth" of a proposal to the re- 
search organization. It has been mentioned previously, and deserves 
re-emphasis, that the measure of value is a function of objectives. 
Thus, the rules for value assessment must be known before assignment 
|is made. 

To illustrate the nature of the value assessment problem, con- 
sider the case of storage battery technology, and the problem of 


assessing the importance of three battery parameters: volume, cost 


ee ee ea 


‘and time between recharging. Assume that the importance of each 


| parameter will be indicated by assigning to each a number ranging 


eee eens 


'from 10, for great importance, to 0 for no importance. Further, 


| assume that an assessment of importance, or value, will be made by 
‘each of the following:! 
1) the user; a Lieutenant, U.S. Navy, commanding a boat con- 
taining batteries and drifting on a Vietnamese river on night patrol. 
2) the R & D manager; an Admiral, responsible for the Navy's 
total research and development program, while considering next year's 
budget. | 


| 3) the boat designer; a naval architect designing a boat for 


use in Vietnam, 





| 
| 
| 
| 


, 
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In, J. Cetron, "Technological Forecasting: A Prescription for- ° 
Ene Military R & D Manager,” p. 29. 
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4) the R & D engineer; a project engineer working in a Navy 
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laboratory and seeking to improve the general performance of batteries. 


Obviously, each of these individuals will consider battery 
‘parameters from different points of view. It should notbe surprising 
if the responses of the four individuals were as different as those 


;Shown in Table l. 


TABLE 1 


BATTERY PARAMETER RATINGS 


Parameter User Manager Designer Engineer 
Volume 3 es 10 8 
Cost 0 10 -2 Z 
Time Between 10 Z 4 i 
Recharging 


If the operational objective of a battery development program 


iwere to provide the best battery for combat units, the value assess- 


. 
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ment of the user would likely be the most accurate. However, if the 


objective were to maintain some battery research and development de- 





|spite budget cuts, the manager's assessment would be appropriate. 
Similarly, other objectives would lead to the selection of either the 
designer's or the engineer's assessment. 

This example should serve to illustrate the important fact 


‘that research and development programs have no intrinsic value or 


.: 


‘worth. Value must be measured with consideration given to objectives. 
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The preceding example also shows a method of translating 
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‘qualitative judgements into quantitative measures. Before conducting 
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-the exercise associated with the data in Table 1, the R & D engineer 
could have related that he considered battery volume to be more 
Significant than cost or time between recharging. This observation, 
however, would be of limited use to someone selecting a battery on 
maeeDasis Of the eEmgineer’s criteria. After the process of rating 

the parameters on a 0 to 10 scale, though, one realizes that the 
engineer regards volume as being four times as important as cost and 
eight times as important as time between recharging. Now, a quantita- 


tive selection is possible. 


Selection 


The selection problem involving alternatives having n signif- 
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icant parameters takes the general form of! 


n ne 
Vk Y Wii = 2) ai Xs (he) 
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the value of the kth alternative 


where v, 
Wy = the value of the ith parameter of the kth alternative 
a; = the weight of the ith parameter 


Xj = the ith parameter of the kth alternative. 


ee 


lereeman, "A Stochastic Model for Determining the Size and 
Milocationw of the Research Budget," p. 2. 
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- This type of selection can be demonstrated by considering 


jthree hypothetical alternative batteries; the parameters of which are 
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shown in Table 2. Applying equation (II-1) to the data in Table 2 (the 


. TABLE 2 


BATTERY PARAMETERS 


Parameter #] #2 #3 
Volume (ft) 80 1.10 95 | 
Cost ($) 110 80 87 | 


recharging (hr) 


| 
| 
| 


Time Between 16 iz 14 , 


‘data being normalized by the mean for each parameter), and using the 


| 


‘weighting values given by the engineer in Table 2, one obtains 


} 
os 80 110 16 . 
Vy = 8X -get expe tl xq 
= 10.26 
v2 = 12.80 
vz = 10.90. 


Since it is presumed that one would seek minimum cost, volume 
and time between recharging, the alternative having minimum value (v) 


would be selected; that is, battery number 1. It should be noted that 


° 
aren ee ee 
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the results of the selection process would have been substantially 
different had other criteria from Table 1 been employed. 
| 


The preceding discussion illustrates the essential character- 


istics of the selection problem. First, the problem involves numerous 











alternatives; say n. Each of these has a present net value, V5. 


Though not mentioned explicitly, before, each alternative will also 


have an associated cost, C;. If one assumes that alternatives are ! 


independent, and that the V5 and, are sufficient estimates of actual 
a 


: . | 
max ) V3 (II-2) 


values, then the general selection problem is to simply find 
1=] 
| 


where V; = f;(Cj) . (II-3) 


to simply selecting all alternatives. 1 
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A constraint that is commonly encountered in selection efforts, 


Unconstrained, the solution of this problem is trivial since it amounts 
is that which limits available funds, in the form of a budget. In 


this case, for T years in the planning period 
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it (II-4) 


OQ 
~ 
" 
1) I eet Her | 
a 
Q 


where c;, is the cost during year t. If an annual budget, B,, must 


be observed, then the solution is subject to 


n 
jepuve rec yp at = 1, ..., 7 . (II-S) | 
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ape Economy Division, American Society for Engineering a 
Education. The Fourth Summer Symposium Papers (Hoboken,: New Jersey, 
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where. u; = 1 if alternative i is selected, 


0 if alternative i is rejected. 
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Interdependence 


The general selection problem, described by equations (II-2) 
through (II-5), can be expanded by admitting interdependence among 


alternatives. This provision would result in | 


V3 = fi (C7, Rees C;, eres oo) ;: (II-6) | 


that is, a given alternative may be affected by expenditures for other 
alternatives. The most common example of this situation arises when 


two or more different research projects are being conducted by the 


same working group; for example an organic chemistry branch. In 

Situations such as these, though the research projects are different 

in substance and structure, by virtue of the mere fact they all relate 
:° organic chemistry, it would be expected that some interdependence 


would exist. 
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As soon as the possibility of interdependence is admitted, 
the nature of the selection problem changes substantially. Though 
the objective of maximizing expected value, as expressed in equation 
(II-2) is still applicable, it is no longer a simple, or trivial 


problem. Recognizing the implications of equation (II-6), which can 


be rewritten as 











where @ is the transition matrix, relating C to V, it becomes apparent 


that some techniques, other than simple rank-ordering, will be re- 
quired to obtain a solution to this selection problem. 

The first step in treating the consideration of interdepen- 
dence is to estimate its quantitative characteristics. The first 


major attempt to do just this was made in Project PATTERN. | This 


project sought to examine the contribution of so-called "cross-support" 


among research projects, and to use the information, thus gained, in 
planning future research efforts. 

In the PATTERN report, an example research program, a 
Strategic Manned Base Orbital System, was compared with three other 
related projects. The objective of this comparison was to determine 
the progress to be realized in the Orbital System research if success 
|were achieved in any one of the other three Brovectswe ine results of 
the cross-support analysis are shown in Table 3. 

The data in Table 3 relate to certain elements of the Orbital 
System program (for instance, elements 4, 32, 33, 106, and so forth). 
For each of these elements, the degree of correlation with the re- 
search objectives of the three other projects being examined are 
shown. For example, element number 123 of the Orbital System project 
had a .05 correlation with elements 1531 and 1532 of the Infra-red and 


Radar Ranging project. That is, if the objectives of the Infra-red 


IaAaron Eeestice, project Patter, Presented to the Joint Na- 


tional Meeting, Operations Research Society of America, October 7, 1969 
(Minneapolis, Minn.: The Institute of Management Sciences, 1964), p.15. 
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TABLE 3 


CROSS SUPPORT ANALYSIS OF 
STRATEGIC MANNED BASE ORBITAL SYSTEM 


Manned Hi Scan Rate Higsean Rate Hi Scan Rate 
Base IR and Radar Laser Range Radar (10-30 KNC) 
Orbital Ranging and Angle (1 MV Peak) 
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Source: Jestice, Project Pattern, p. 15. 











project are achieved, then .05 of the effort to achieve Orbital 


System objectives would be unnecessary. 

The significance of cross-support, or interdependence, among 
research projects is readily demonstrated. For example, assume that 
the Orbital System project has been selected for funding. Now, con- 
Sider the problem of selecting one additional program, from among 
the three alternatives shown in Figure 3. If it is further assumed 
eat the objectives of the three alternatives are of equal importance, 
and that the independent costs are the same, then, obviously, one 
would select the Laser Range and Angle Radar project for funding. 
This is so because the actual costs will be substantially less than 
the independent costs due to the degree of interdependence indicated 

| by the data of Table 3. 

It should be noted that the measurement of cross-support, as 
developed for Pattern, is a technique for quantifying the interde- 
pendence of research projects by examining the direct relationships 
among the projects. Another approach to the problem of accounting 
for interdependence is to measure project relationship to some ob- 
jectives other than the interdependent projects. A typical applica- 
tion of this type of analysis is to examine the contributions of a 
given project to a number of selected technological disciplines. 

One method for measuring project content in this manner is 
that of PROFILE (Programmed Functional Indices for Laboratory Eval- 


1 . : 
uation). This procedure, developed for evaluation of military 


erick Jantsch, Technological Forecasting In Perspective (Paris: 


Organization for Economic Co-operation and Development, 1967), p. 228. 
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research projects, relates project content to various military missions.: 


| 
! 
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The type of analysis utilized in PROFILE is an analysis of program 
relevence to prescribed objectives. 

Relevence evaluation is most easily described by constructing 
a matrix to display the form of the analysis. ! As shown in Table 4, 
each be the n program alternatives is related to the objectives, «a 


through v, by the significance numbers, s. In addition, each of the 


relative to the other objectives. 


| objectives is assigned a weight, q, which indicates its importance 
! 

t 

| The desireability of research programs can then be measured 


by relevence numbers, r, which are obtained from 


ee =. 


Vv 
as G y Gaeya _— ek 
X=a 


To ensure the homogeneity of the selection logic, the data in Table 4 


is normalized subject to 


Vv 
=a s 


1 


Ibid, p. 221. 
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TABLE 4 


RELEVENCE EVALUATION MATRIX 


Weights Program Alternatives 


Objectives OF 
Objectives 


Source: Jantsch, Technological Forecasting in Perspective, 
De 2c). 
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Summary of Evaluation Techniques 


The process of selecting research and development programs, as 
it has been described, is one of sequential decision-making. However, 
too frequently, the sequential aspects of such an endeavor are not 
recognized by the research and development manager. These sequential 
characteristics are of prime importance in the description and struc- 
fmeuring of the objectives and criteria to be used in project selection. 
\The primary reason for selecting particular projects from those avail- 
lable is to maximize the ultimate benefit to the organization and to 
maintain some assurance that research efforts will be productive. As 
long as piectives, and the relations among them, remain unclear, it 
becomes doubtful that any selection method will achieve much success. 
A common failing of selection techniques lies in a disregard for 
proper planning and analysis in the formulation of selection systems. 

The preceding description of project evaluation and selection 
principles has been presented in terms of the sequential decision- 
making process. Figure 2 displays the activities involved and the 


manner in which they are related. The description of the flow of 


lpaker and Pound, "R and D Project Selection," p. 131. 
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FIGURE 2 


EVALUATION AND SELECTION PROCESS 
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FIGURE 2--Continued 
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activity is intended to emphasize the serial nature of the process. 


It should be emphasized that, once operational objectives are estab- 
lished, no further feedback function is encountered until the eval- 
uation/selection exercise. The significance of this lies in the 
understanding that modifying objectives for the purpose of making 
particular programs acceptable, defeats the intent of an organized 


selection effort. 


Implementation 


The selection of research and development projects is essen- 
tially a planning function. It commences with long-range plans and 


forecasts, and results in short-term plans for meseanch CLLOres a Une 


‘| next step, then, in funding of research and development, is the 


| process of implementing the selection decisions.! 


| Time and Cost Parameters 

The two most obvious characteristics of a research project, 
other than its subject matter, are the cost of the project and the 
time until completion. The importance of cost is obvious, for the 
undertaking of an organized selection effort is RAEN by a need to 
allocate limited, financial resources among numerous candidate re- 
search efforts. Cost has added significance for those programs which 
span more than one budgetary period (usually a year). This results 
from the fact that monetary contraints are usually based on the dollar 


‘Souder, "Experiences with an R & D Project Control Model,": 
p. 40. 
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jbasis, with available monies being increased considerably, in antici- 


requirements of research programs for a given‘period, rather than the 


dollar requirements over the life of the project. In this regard, then, 
the research and development manager is usually interested in the dis- 
frabution of funds over the life of the project. 

Project cost can be measured in numerous ways, and the method 
to be used is strongly a function of the nature of the organization 
conducting the research and development. In most commercial enter- 
prises, for instance, a project incurs not only direct costs for 
material and labor, but also period costs and opportunity costs as 
indicated in Figure 3.1 | 

One example, however, of research and development funding 
which is not constrained by annual budgetary limitations, is govern- 
ment programs, which are the subject of this paper. Government re- 
search programs are generally selected on the basis of total cost with 
allowance made for the fact that appropriate annual fund appropriations 
will be forthcoming. | 

Another aspect,of cost distribution, other than that con- 
cerning period budget constraints, addresses the question of 
efficiency in the rate of expenditure. At the root of this consider- 
ation is the fact that research achievement is not a linear function 


of funds expended. At times, research programs are funded on a "crash" 


pation of realizing earlier success. Though project completion may be 


Rosenbloom, "Notes on the Development of Network Models for 
Resource Allocation in R §& D Projects," p. 60. [ 
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FIGURE 3 


PROJECT COST FUNCTIONS 
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Source: Rosenbloom, "Notes on the Development of Network Models 
for Resource Allocation in R & D Projects," p. 60. 








expedited in such a fashion, there exists an inherent wastage in this 


type of effort. This wastage will result primarily from less 
effective control and co-ordination and less efficient utilization of 
learning processes. Thus, wastage will occur when expenditures exceed 
some "satiation" level, as igeanccd ineChapter I. 

One study, of the expenditure-achievement mechanism, found 
that achievement, as a function of expenditure, approximates an ex- 
ponential function. 7 This sani hypothesized that there exists, for 
ithe ith period of a progran, some expenditure, y;, which can be con- 
Becerea efficient. Then, if x; is the actual expenditure for that 
period, the effective expenditure, z;, or that expenditure which 


actually contributes to achievement, can be found from 


N 


ae 4 


i= Yi Yj Sieaene i S 


we 


where ¢€; is a characteristic of the research project. 

A final aspect of the study concluded that wastage, adherring 
to inflated expendi ture rates, was more pronounced during the initial 
stages of a research program, and decreased in significance as the 
effort matured. Initial activity in research programs is generally 
devoted to seeking direction and definition. Consequently, added 


Jatkinson and Bobis, "A Mathematical Basis for the Selection 
SE Research Projects," p. S. 
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funds made available during this period will most likely be applied to 


Paraile? exploration. The results of these funds, ultimately, will 
not be germane to the research effort and therefore, must be con- 
sidered wasted for purposes of the project. However, the acceleration 
of funding in the latter stages of a project will result in parallel 
endeavors in more closely defined areas, such as development, and will 
be more productive in terms of useful results. 

Another study of considerable interest, in this regard, was 
that conducted by Hess, as described in Chapter I. As a result of 

-s study, which examined the optimal expenditure for a given ee 

of a project, and gave consideration to the history of preceding ex- 

‘penditures, Hess concluded that, in general, actual expenditure 

| practices closely approximated the theoretical optimum. + One can con- 
clude from this, that the prescriptive, or intuitive practices, 

a are still the foundation for most research and development 
funding procedures, are substantially adequate. Indeed, one study, 
which Ponined precisely the question of subjective forecasts and 
estimates by research and development managers, found that Srescineiven 
practices are not unreliable, and the concomitant subjective forecasts | 
can be extremely accurate. 

‘From the foregoing, then, one must recognize that expenditure 


rates in research programs, that is the distribution of expenditures 


over the life of the program, is worthy of consideration. However, 


phees. "A Dynamic. Programming Approach to R and D Budgeting 
and Project Selection," p. 178. 











analytic techniques which address this problem are ultimately founded 


on subjective estimates from operating personnel. Further, there is 


every indication that subjective estimates of expenditure rate require- 


ments are quite adequate. 
The second significant parameter of a research project, that 


of time until completion, has two important areas of impact in the 





is, as continuation of a project is experienced, the cost of maintain- 


y 
| s ° e e e 
ing the research team and providing services is encountered. Second, 





land probably more important, is the decreased utility of the results 
of the research due to the delay. In commercial enterprises this may 


|be realized in diminished, competitive position. In military re- 


— 


‘search and development programs, the results may be of even more 





‘Significance, such as the diminished strategic position which has 
Ftc from the delayed development of the F-111 fighter-bomber. 
The immediate consequences of extended time in research, 
though, are usually subordinate to those resulting from extensions in 
expenditures. Most organizations operate within well defined limita- 
tions on financial resources. On the contrary, time available is of 
infinite supply. 
| Quantitative techniques devoted to the consideration of the 

time factor in research are usually limited to estimates of expected 


value; intrinsic or commercial. One study, for example, estimated 


that the commercial value of the product of research would decrease 





value of a project. First is the added direct cost due to delay. That 
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by some factor, k, which is derived from projected sales patterns, * 


Therefore, if Mi is the original estimate of commercial value, based 
on an assumed completion date, the value after n years of delay would 


be 


aa-mce(enek)™ . 


Summary of Implementation Procedures 


Time and cost in research programs are quantities that are 
easily measured and readily incorporated into conventional control 
procedures. These parameters, then,-are the essential character- 
istics of programs which are considered in most selection processes 
and also form the basis for program control. 

Despite the utility of time and cost as primary program 
characteristics, difficulties encountered in the administration of 
research and development programs indicate that these parameters may 
not be wholly sufficient for the purposes intended. To this extent, 
then, the following discussion will examine features which should be 


added to control procedures. 


Achievement as a Parameter 
The ultimate-intent of any research and development effort is 
the realization of some process, procedure or product. The success 


of the effort is judged by a comparison of the actual results with 


Latkinson and Bobis, "A Mathematical Basis for the Selection 


BieRescaneh Projects,” p. 3. ; 
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those intended. Planning and control activities associated with re- 


search and development, assume, in the traditional manner, that given 
research efforts can be characterized by measures of time and cost 
which can be associated with the subject research and its intended 
results. As was discussed in the introduction, however, notable 
failures in the control of research programs have been experienced. 
This usually results from a lack of correlation between research 
achievement and estimates of associated time and cost. 
The ineffectiveness of many control systems can be attributed 
| to a disregard for the relationships among achievement, time one cost. 


It should be apparent that an effective control system must yield 





‘valid indications of the ultimate results of a research program by: 
informing the manager of any deviations from planned values of 


|achievement, time or cost; indicating corrective actions needed; and 


communicating these indications to the manager. To this extent, then, 
an effective control system must merge achievement with time and cost. 
However, most control systems, utilized today, separate achievement 
reporting from cost and time accounting and, therefore, contribute 

to ineffective performance. - 

Numerous. reasons might be suggested to explain the observed 
disinterest in achievement measurements or the difficulties en- 
countered when such measurements have been attempted. One of these 
reasons is that it is frequently difficult to analyze achievement. 


For some development projects, achievement indices based on quality 


1 


tomes acetmmeme te 


| 


| 
} 


! 
Souder, “Experiences with an R & D Project Control Model" p.40.' 











and quantity of accomplishments can be devised. However, most re- 


search projects are not as easily handled because the significance of 
interim findings may not be known until the project is nearly com- 
plete.? Consequently, the basic mechanics involved in implementation 
are often the source of discouragement. Many of the difficulties 


encountered in attempting to quantify progress in research and de- 


ee ee eee tee, 


panopment result from a literal-minded approach to the problem, which 
is reminiscent of conventional production-type operations. 
| As a result of this neglect of achievement in control systems 

an inordinate amount of significance is generally attached to project 
. overruns. This, in turn, contributes to the introduction of 

Babee istic reporting of results, frequent neglect of projects which | 


‘operate within the budget and the tendency of project managers to 


expend funds, regardless of need, in order to avoid budget re- 


| 

Huctions in following periods. 
| In summarizing the status of management science, as it 

applies to research efforts, two salient features warrant emphasis. 
First is the practice of selecting and funding projects primarily on 
the basis of qa prtort data; that is, on the basis of initial estimates 
Seecost, time and utility of the product. This, NeNEV ED: is a pe- 
culiar anamoly when one considers the uncertainties inherent in re- 
search and development. The longevity of decisions regards selection 


and funding seems quite inappropriate in view of the tenuous nature 


of the informational basis for these decisions. 


libid., p. 42. 











The second feature of the process concerns the nature of the 


inaccuracies in data. The estimates required by conventional se- 
lection and funding procedures are based on a manager's ability to 
anticipate the performance of personnel in research efforts. How- 
ever, in many research programs ie nature of the forthcoming work 
is Enknown. Therefore, it is virtually impossible to ascertain 
whether the talents and capabilities of a research team are appro- 
| priate for the task. Even though estimates of costs for equipment 


land salaries for a project may be reasonably accurate, one cannot 


J 


determine, on an @ prtort basis, that these resources will be used 
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efficiently and yield expected results. 
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CHAPTER III 


CONSTRUCTION OF THE MODEL 


Having examined the more important ingredients of the problem 
of program selection and control, attention will now be focused on the 
‘procedures which are advocated by this paper. The procedures and 
| techniques, which have been selected for use, constitute the model 
which is the basis of the quantitative method. The two major com- 
ponents of the model, dealing with selection and control, are dis- 


| 


‘cussed separately for they are two relatively distinct operations. 


The process of project selection, as the preceding discussion 


j 
RECVece Selection 
4 
| 


has related, and as is shown in Figure 2, involves the measurement of 





project utility with consideration given to established objectives, 


and taking into account the effect of interdependence among projects. 





As was pointed out, quantitative techniques are not employed 





to any extent until the formulation of operational objectives. Con- 
sequently, this discussion will assume that preliminary planning has 
been accomplished to the extent of having established operational 
objectives. Therefore, what remains to be shown is the method of 


completing the selection process on the basis of these operational 


objectives, project utility, interdependence and certain aspects of 
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funding. 





Utility Analysis 
In selecting a quantitative method for utility analysis, one 
must first decide how interdependence is to be incorporated. This 
does not result from any particular theoretical requirement, but, is 
simply a starting point which facilitates construction of the model. : 
The preceding chapter discussed two approaches to interde- 
Bae Deence ; that used in PATTERN, which is based on the direct measure- 
| ment of cross-support among projects, and that used in PROFILE, which 
uses indirect measurement methods. Either of these approaches are 
adequate for purposes of project selection. Therefore, the basic 


| technique of the PROFILE approach is adapted since it is more easily 


implemented. 





| ; The PROFILE approach relates Bee ee re naiseienistics to re- 
mecatch objectives. However, in actual application situations at the 
laboratory level, the relationships between projects and general 
objectives may be confused by intermediate requirements. Moreover, 
relating project characteristics directly to general objectives, which 
are usually strategic and of managerial nature, does not correspond 


to the process described in the preceding chapter, and shown, 


= see Owrec sees 


graphically, in Figure 2. Therefore, the method developed here will 
utilize requirements prescribed at the laboratory level in accordance 
With operational objectives. 


- * | 


The process of relating research projects to research re- 


quirements, for the purpose of project comparison, requires that the 
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subject requirements be expressed in terms that are applicable to all 


: projects; otherwise the continuity of the comparison would be destroyed. 


In order that this requirement of continuity may be realized, an 
‘added measure is interjected. This added measure is expressed in 
‘terms of scientific disciplines which relate to both projects and 
feetarenents.- The desired relationships are then obtained by de- 
Semiving requirements in terms of the scientific disciplines, or 
technologies, which must be supported, and by describing projects in 
j terms of the technologies to which they relate. The method de- 
scribed is shown graphically in Figure 4. 

Pescescribe tie process of relating projects to sciences, con- 
@eacer three projects, Pi, P, and Pz, which contribute, in various com- 


binations to four sciences, S}> S5, Sz and oe Next, assume that Py 


supports Sy and S5> Pa supports S., S3 and S,, and P. supports S, and 


Sz. It is now possible to quantify these relationships by con- 


structing a transfer matrix 
ae AE 


where 


= J 
aij = 


Shee ee ecGes mOt Support 5. . 


| 1, of P, supports S;; 
| j 





| Aa, v. bean, = V. Dean, "A Research Labortory Performance Model," IEEE 
Transactions on Engineering Management, EM-14 (March, 1967), pp. . 44-45, 
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FIGURE 4 


DIAGRAM OF PROJECT, SCIENCE 
AND REQUIREMENT RELATIONSHIPS 
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Mrethe project-science relationships are presented as in Table 5, it 


is apparent that 


he=noe 1 1) «= (III-1) | 
1 01 0 | 
| 


The relationships between sciences and research requirements 


can be similarly described using the transfer matrix | 
B= (054) | 

where | 
; 


1, 1£ a development in Sj is required to satisfy R,; 


bb... = 
Hs 0, otherwise. 


If the science-requirement relationships are assumed to be 


those shown in Table 6, then 


OM 

B = (1TI-2) 
le Ol 
OES ame 


On the basis of the preceding analysis, it is now possible to 
determine the relationships between projects and requirements, which 
was the original intent. Recall that, if project 1 contributes to 


Selence j, then 


en ee ee 





TABLE 5 


PROJECT-SCIENCE RELATIONSHIPS 





Sciences 


Projects Po 





TABLE 6 


SCIENCE-REQUIREMENT RELATIONSHIPS 


Requirements 


Sciences 











' ai; = l, 


Likewise, if a development in science j is necessary to satisfy requ- 


irement k, then 


b Ms 


jk © 
Consequently, if project i ultimately contributes to requirement k, 


then 


ai 5>5K = Le 


Likewise, if either 





ae = 0, or bak = 0 


then project 1 is not related to requirement k, and 
It should be observed that a given project can contribute to 
some requirement in more than one manner. That is, a project may 


: 
| 
i 
t 
| 
| 
j Support several different sciences, which each contribute to the same 
t 


Bagdeezorent. For purposes of selection, it is necessary to determine 
the number of ways in which each project does contribute to each re- 
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quirement. If c;, is identified as the number of ways in which project | 


i supports requirement k, the C;; can be computed from 


Cik = ) 45> 5x : 
3 


If C is identified as the matrix of Cie then 


C=AB 


To illustrate the application of this analysis, consider the 


values of A and B given in equations (III-1) and (III-2). For these 











values, 


C=AB 
IO ane 
18) 
Ot 1 
ae O etal 1 
gai Dineel 
em.) .C ; 
el eae 
or 
ete 1 
eee 2 2 3). (111-3) 
20 1 


| The significance of C can be shown by an examination of some of the 
elements, c3,. For instance, 
aleecl2)> <5 > 1 
indicates Ciateprovectsl Contributes to all three requirements in one 
way. The fact that 
C5, = 2 
however, indicates that project 2 contributes to requirement 3 in 
three different ways. This is, indeed, the case, via-sciences 2, 3 
.{ and 4, 
The structure of the model, ans ean and the method of 
analysis employed, enables the ready determination of project utility, 


to the extent that the utility is a function of the number or require- 
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ments supported. This, however, is seldom the criterion, for some re- — 


quirements are usually more important than others. Indeed, this 








62 





would likely be the case when considering operational objectives. 


When objectives or requirements are examined, it is fairly 
easy to rank them in an order of importance or desireability. This 
was demonstrated in the preceding chapter when the parameters of stor- 
age batteries were considered. ise Gk several rating techniques may 
be Eiilized, depending on the nature of the requirements being con- 
Sidered. Regardless of how requirements are assessed, the result will 
be a value function, V. Assume, for example that a research admin- 


jistrator had examined the research requirements Ri» Ro and Rz, and 


rated each on a 0 to 10 scale, and that the results of his evaluation 


~ 


were 


v (Rj) = 10, 
v(R>) = 2 
and v(R3) = 4. 


~ 


| In Onder to ensure the uniformity of the selection logic, these 


ratings must be normalized to satisfy 


) Vj =k. 
i 


After normalization, it is possible to determine 
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Having determined the value of the research objectives or requirements, 


it is then a simple task to evaluate the research projects. Recall 


that C, as shown in equation (III-3), relates projects to requirements ~ 


in terms of the number of requirements supported. Having determined V, 


one can then ascertain the value of projects, Tr; 


of the several objectives, v;, from 
‘Sil 2 Ms (111-4) 
This, for the example used previously is found to be 


I 


1 ; : 
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which shows the relative values of the three candidate programs. 
Success in using analysis techniques, such as those employed 
iim nOriLE, 1s dependent on the degree of orderliness achieved in the 


process of determining the possible combinations of project-science 


and science-requirement relationships. In order to illustrate an 


approach which has some likelihood of success, a method developed for 
the Naval Material Command, Washington, D. C., is presented in 


Appendix A. 


Expenditure Analysis 


After having computed the value of alternative projects, it 


is necessary to examine project funding so that a cost-benefit 


, in terms of the values 


= Aa eee 
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|comparison can be conducted. ‘The primary interest, in this regard 
‘lies in the relationship between expenditure and achievement. When 


ithe research manager is contemplating the allocation of funds, this 


: Peiationship 1s needed to determine the desired level of project 


Over time, the distribution of expenditures for research and 


' 
| 
- 
| 
= 
|development projects, is generally expected to be similar to one of 
| 1 


two likely patterns. The first typical pattern shown in Figure 5a, 





is the decaying expenential; approximated by 

mix) =e PiT1=6) 
This type of distribution characterizes a project which commences with 
large initial outlays, with a subsequent decline in expenditures. [he 
second typical pattern, shown in Figure 5b, represents projects which 


i ‘ 

| commence more Slowly, usually in some small exploratory effort, and 
| 

| 

t 


build up to some maximum expenditure which is then followed by decline. 


Although these expenditure distributions are of interest for 


i budgetary use, it is the cumulative distribution which is of greater 


interest for purposes of project selection. As indicated in Figure 6a, 


the cumulative distribution, corresponding to the expenditure pattern 


a ee 


Teehwgire oa, 1S represented by a function of the form 
PCa Ice er 4 (iio?) 


Similarly, the distribution of Figure 5b relates to the general S 


1B, V. Dean and S. S. Sengupta, "Research Budgeting and 
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FIGURE 5 






TYPICAL EXPENDITURE DISTRIBUTIONS 
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Source: Dean and Sengupta, “Research Budgeting and Project 
Sevecerion.,  p. 169. 
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FIGURE 6 





CUMULATIVE EXPENDITURE DISTRIBUTIONS 
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Source: Original; derived from integration of distributions in 
Figure 5. 











67 





‘curve shown in Figure 6b. The distributions, however, relate cost 


to time. What is actually of interest is the relationship between 
'cost and achievement. 

| The impact on project achievement of variations in funding 
“can be determined by considering the progress of a typical project for 
Bei ferent PmMcimMegeLevers. srigure 7 Shows project progress for four 

! 

different funding levels. The four levels illustrated commence with 
Fieve! 1, which is one-fourth of the satiation level, M, and increase 
“in one-quarter increments until M is reached. 

| To determine the cost-achievement relationship, one simply 

1 

rakes a given time, t,, and compares cost-achievement points for that 
iaiee. Tc ee of such an examination are shown in Figure 8. As 


is indicated, for the achievement-time relationship of Figure 7, the 


resulting cost-achievement relationship is the familiar exponential 


ee ne nee eet me tame ee) eee 


as given in equation (III-7). However, the cost-achievement function, 
under other Spemeeneas, may be similar to Figure 8b. The ultimate 
conclusion, then, is that achievement, as a function of cost, is 
Similar to either the decaying exponential, or the S-curve.! 
Quantifying the cost-achievement function of research projects 
can be simplified even further. As shown in Figure 8, the two types 


of distribution are essentially the same above the achievement level 


. 
Ane ee ew etn ow we a= Sere ote care ene ORD 8 oe eeteceteent ane eves tems) we te en Seer hy te: 


corresponding to critical cost. Since management is not likely to 
be interested in funding levels below the critical level, the two 
Iw. rE. Souder, "Planning R & D Expenditures With the Aid of a 


Computes, —budpeting, Mareh, 1966, p. 27, or Ayres, Technological 
, Forecasting and Long-Range Planning, p. 177. 
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FIGURE 8 






COST-ACHTEVEMENT RELATIONSHIP 
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Source: Souder, "Planning R & D Expenditures with the Aid of a 
Computer," 21, 
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I siceton Canepe modeled by the expression for Figure 8a, as 


e 


given in Equation (III-7). 





Having determined a form for cost-achievement, it is next 


Ph pean 


ee 


necessary to find the parameters for the selected exponential. As 
would be presumed, the exponential is a function of the proposed fund 
eiskocation for project i, x, In keeping with convention, x; is 


normalized by the function's time constant, 


| 
| 
| 
| 


Ril 


> 


which is that value of X; at which 63 per cent of maximum achieve- 
ment is realized. This notation is shown in Figure 9. 


Since the functional form of cost-achievement has been 





Uniquely specified, the relationship for any particular project can 


| 


t be found explicitly once any single data point is obtained. ‘fhe 


procedure for this becomes apparent when one realizes that the 1.0 
achievement level in Figure 9 corresponds to maximum (or unlimited) 
i funding. Consequently, achievement is measured relative to the 


possible maximum. An achievement estimate can then be obtained, for 





a particular proposed expenditure. By examining a research forecast, 


. 
aaa 


it is possible to compare the expected advance in the achievement 





| parameter, for the proposed expenditure, with the advance that might 
be expected from a maximum expenditure. Figure 10 shows the appli- 
; cation of this method for a research project concerning high strength 


e 1 e e ° 
resins. FOoreias research project, then, an achievement-cost curve 


aes Department of the Navy. Naval Applied Science Labora- 
tory, Technological Forecast in Depth: Organic Materials, 1968-1988, 
Pabeeeesccec J50-100, Final Report (1968), p. 18. 
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would be determined by plotting x; on the abscissa, ays /ay on the 


ordinate, solving for t from 


and completing the curve for the value of t thus obtained. 


| 


Optimization of Selection 


As the result of utility analysis and expenditure analysis, 
each project can be characterized quantitatively by combining 


equations (III-4) and (III-7) to obtain a profit function for each 


ne ee ee eo ee eee ne eee 


‘project in the form 
Xi 


oF 
ppm d-e °) (I1I-8) 
The objective of the optimization procedure, then, is to maximize 


a 


‘subject to a padeer constraint of 


ih 


| 

| 

| 

| 

| 

=) pj (III-9) 

) x} ce 0. _ (III-10) 
i 


Presuming that the intent is to expend the budgeted amount, this 


constraint becomes 


= ) ce, 2 0 . Giy=11) 
i 


The task of optimizing equation (III-9) can be accomplished by using 








1 To implement this 


the technique of the Lagrange multiplier. 


technique, let 


= P - XB 
Xj 
, [ vate ncaa "hy - AX; J (III-12) 
i 





where X is some constant (the Lagrangian multiplier). Now, since Q 


differs from P by only a constant, AB, then, a maximum for Q will 


oo 2a e EE aw oiewing tes =o. 


correspond to a maximum for P. To find the maximum for Q, compute 


8Q 
OX; ~ 
xi 
mie) = 6 (lan 
THF . 


: 
| 
| 
! 
| 


2st 
2 era 
EQ 2 Ue ae, 
ox? rf 


cat 
See nS. 
5 aes (III-14) 
Ti 
Tif 
y 480 = 1s 
X5 a tn ity (III-15) 


This is ensured to be a maximum, since 


ly. S. Sokolnikoff and E. S. Sokolnikoff, Higher Mathematics 


for. Engineers and Physicists (New York: McGraw-Hill Book Company, 
Inc., 1941), pp. 163-67. . 











Now, aS was provided in equation (III-11), the only solutions to be 


admitted are those for which 


Xe 


pee 


This in turn requires that the argument of the logarithm in (III-15) 
be > 1. These constraints can be re-stated by acknowledging that for 


anyen, either 
x. = 0 (T11216 41) 


or dh < Zi (III-16.2) 


To explain the Significance of equation (III-16), assume that of a 
i total of n possible projects, some sub-set, o, of the n, constitutes 
those projects which will be funded for optimum return. For each of 
the projects in o, since x; is non-zero, then 


{ 
| 
i 
| 
| 
| 
| 
! 
| 
| 
| 


‘eae Ome 
My 


Ofecourse, for all projects not in o 


x, = 0. 


In view of this, equation (III-11) becomes 


Bel x, . (III-17) 
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‘X = exp oe (111-19) 


a 


MemlseeVicentetnac the value Of 9A is a function of the projects 


optimization procedure, some approach is needed which avoids the 
Nomreaice the serrorte required in seeking o, the following 
algorithm can be employed. First, the candidate projects must be 


arranged in descending order of the cost-benefit factor, 


ae nS ee ORS Sm EUS UE Rs to ece 





In other words, arranged such that 


| Bes as > (111-20) 
2 a2 2 


Next, o is assumed to consist of project 1 and project 2, and d is 


computed from equation (III-19). Then, equation (III-16.2) is invoked 


selected for inclusion in o. Consequently, in order to implement this 


necessity of considering the multitude of possible combinations for o. 


fori=1andis= 2. If equation (III-16.2) is satisfied in each case, — 


Encneouloeexpanded to include project 3. The subset, o, 1s 








Cell 


: 
| 


i progressively expanded, with A being re-computed and checked at each 
‘iteration. The iterations cease when a project, k, is introduced into 


| the subset such that equation (III-16.2) is violated. Having reached 


—~ 


this point, it is determined that o consists of the first k-1l projects 


sree Vues or CS a a oS 


in the ordering of equation (III-20). Then,-A is computed from 





i 


!equation (III-19), including the selected k-1 projects. Finally, 


ot 


for this value of A’, the fund allocation for each of the selected 





projects, Xs is determined from equation (III-15). This algorithm 


is shown, in flow-chart form, in Figure 11. 
Should o consist only of project 1, the combination of 


equations (III-15) and III-19) reveals that 


as one would expect. If, then, project 2 is added to o, A increases 
as per equation (III-19). Likewise, x} is decreased to some value 


| 

i 

| 

‘ 

| less than B, with the balance being allocated to xp: 
xX, < B 
X, = B- xX - 

| 


As X} increases with each addition of a project to o, it is compared, 





via equation (III-16.2), with successively smaller values of 


TT 
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FIGURE 11 


SELECTION ALGORITHM 


Equation (III-20) 


Equation (III-19) 


Equation (III-16.2) 








FIGURE 11--Continued 





Equation (III-19) 
| 
} 


. 
Source: Original 
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r 
Consequently, if one were to plot the values:‘of \ and the limiting a 
1 


for the series of o's, the resulting locus as illustrated in Figure 12, 


| 
: 


| 


EROVeC t Control 


would demonstrate the determination of o. 
‘Chapters I and II examined various aspects of management plan- 

ning and control as they relate to the administration of research and 
| development aGrivitetes. AS was illustrated in Figure 1, the essential 
eredients of the control process ern comparison of actual and 
anticipated performance and an analysis of observed variances. Though 
= Geseription Of control is, at face value, no different from 

, control in conventional management systems, the significance of con- 
trol in research and development activities, in terms of its im- 
portance for effective and efficient operations, is substantially 
peter. As has been mentioned before, control in most enterprises 
relates to budgetary control of operations. Traditionally, this type 
of Ponto has been successful. The degree of success that has been 
realized is largely attributeable to the Aicce correlation between 
resources expended and results realized. Such a nexus is obvious in 
practices which include instruments like standard cost schedules 
for.production activities. Even less structured business operations, 


such as advertising, are ammenable to analysis on the basis of results 


realized from given expenditures. 


Ioouder, "Experiences with an R & D Project Control Model," p.32 
2 Jay W. Forrester, Industrial Dynamics (New York: The M. I. TA 

Press of the Massachusetts Institute of Technology and John Wiley & 

Sons, Inc., 1961); pp. 199-207. 
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As was discussed in Chapter II, personnel associated with re- 


search and development activities have, in several studies, demon- 
Strated an ability to estimate anticipated expenses. However, other 
Similar attempts have resulted in notable failure. The essential 
characteristic of research and fevelonnent which prompts this dichotomy 
is fhe uncertain correlation between the activities for which cost 
estimates are prepared and the research advances which are expected 
to result from these activities. The supervisor of a research team 
may be quite capable of estimating costs for a proposed project. oR 
ever, the accuracy of his estimates, regards the effectiveness of his 
organization in persuing and achieving technological advance, may be 
inadequate. } As a result, control, in research and development, must 
monitor not only expenditures, but, also, the achievements accruing 


from these expenditures. 


: 
| 
: 
: 
, 
i 
| 
: 
| 
| 


The Role of Prior Estimates 


ae en me oe see - ee 


A prior estimate is a statement of anticipated circumstances 
associated with some future effort. The fact that prior estimates 
concern future efforts is essential, since the estimate is presumed 
to exclude any information of the actual prosecution of the StLont. 

Im thas respect, then, an annual budget is a prior estimate since it 
concerns activities which are yet to commence. | 


Prior estimates are common to the vast majority of managerial 


endeavors. Besides financial budgets, such estimates include production 


Myantsch, Technological Forecasting In Perspective, pp. 53-60. | 
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schedules, personnel projections, material schedules, and the like. 


These estimates derive from objectives which are the results of plan- 
ning. Most such estimates, for conventional commercial concerns, are 
fairly objective from the stand point that the estimates relate to 
Metavities of very similar nature. Therefore, these estimates are 
frequently obtained by using simple, oetneneleesion techniques. 


Estimates for research and development activities are also 


jone of the most common techniques employed in forecasting, or esti- 
a ol | 
mating, is that of trend extrapolation. The use of these common 
extrapolation techniques presumes the existence of a substantial a- 


‘obtained, in most instances, by extrapolation techniques. Indeed, 
| 
i 
| 
| 
4 
| 
,mount of continuity between the past and the immediate future. As 
i 
t 


,has been emphasized in preceding discussion, such a presumption, re- 
| 


| lating to research and development, is inherently uncertain. Specifi- 
| 
| 1). What are the criteria for choosing curves to extrapolate? 
{ 

2)| When is it safe to extrapolate a.rate of change naively, 
‘and when must points of inflexion, or changes in the rate of change, 

2 ; 

'be anticipated? — 
These are two questions that are not easily answered. - 


In attempting to minimize the impact of poor results from 


lames R. Bright, Technological Forecasting for Industry and 





Government--Methods and Applications (Englewood Cliffs: Prentice-Hall, 


Inc., 1968), pp. 56-109. 


2 
Ibid., p. 77. 


, cally, two questions must be addressed when extrapolation is attempted: 


2S OC Eee ee Cen meee 
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extrapolation methods, many management systems, for research and de- 


velopment, implement measures of the uncertainty involved, in the 
form of oaaiantieies of success. However, these estimates are still 
prior estimates, and are usually obtained by some sort of extrapola- 
tion. The difficulties associated with such efforts have been dis- 
cussed - Chapter I. 

The over-riding consideration that must be recognized when ad- 
dressing the topic of prior estimates, is that the utility of such 


techniques, as they have been applied to conventional management 


ee ce ce th gg sh RP 


Systems, is limited. Even attempted modifications, which have been 





|initiated as a result of this limited utility, are difficult to im- 
plement, and frequently have the same inherent shortcomings possessed 
by the methods which they seek to improve. 


From the preceding analysis, one should not conclude that 


Ape a oe EERIE ee aes 


| prior estimates have no valid role in research and development manage- 
iment. On the contrary, they are essential. All quantitative selec- 
tion techniques initiate from prior estimates. It should be obvious, 
that, since selection necessarily takes place before any activity 
commences, only prior estimates are available. Project control, how- 
ever, is a continuing process throughout the life of ie project. Lt 
is reasonable, therefore, to contend that one need not and should not 
mely solely on prior estimates for purposes of controle [he remainder 
of this chapter will be devoted to the development of a method which 


augments prior estimates and affords increased vitality for methods 


of controlling research and development activities. 


ee aes 


Se 











Description of the Process 


The preparation of a proposal for a research project necessari- | 


| 
ly commences with the development of a scenario. This scenario is a : 
time-ordered, episodic sequence of events which are logically related : 
to one another and portray the intended progress of the project. In 3 
informal organizations, scenario preparation may be nothing more than | 


the act of mentally planning the effort. However, in most situations, 


which require the submittal of proposals in a formal fashion, the 


Sete 


; scenario will be explicit in nature and set forth in a published form. 


1 
‘ 
| 
i Procedures for the preparation of proposals, and attendent scenarios, 

| vary greatly among research organizations. The particular approach 

H ; 

| 


described here is one common to some of the laboratories of the U.S. 


i eee 


4 


| To illustrate the development of a scenario, it is advanta- 
| geous to examine a sample project. For this purpose, consider the 

task of developing a technique for the optimal design of super-con- 
ducting electrical machines, intended for solution by electronic com- 
puter. The reason this task would be considered a research and de- 
velopment effort derives from the fact that super-conducting machines 
do not conform to conventional electro-magnetic theory and established 
design procedures have not been developed. mersouet since past design 
and construction endeavors have been limited to a few prototypes, there 


| 
exists no consensus concerning an accepted theoretical model. 


lThe method discussed is one obtained during the period of 
time when the author was Program Officer at the Naval Applied Science 
Laboratory, Brooklyn, New York. 
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A scenario for this research task consists of the following 


events: 

1) Determine appropriate parameters for the model. 

2) Develop magnetic field equations utilizing the parameters 
selected. 


3) Determine operational and theoretical constraints. 

4) Develop design equations. 

5S) Develop optimization method. 

6) Write machine program. 

7) Complete analysis of test designs. 

The initial opinion one is likely to formulate, concerning 
such a scenario, is that it is simply an application of conventional 
scheduling techniques, such as PERT. : Towanmextent, such an observa- 
ficmeis correct, for both serve to indicate project progress, or 


schedule. However, it should be observed that the scenario is a se- 


currently. On the contrary, the events in a scenario can be arranged 
only in serial order. Consequently, completion of a scenario event 
indicates explicitly some degree of project progress. Nonetheless, 
PERT network analysis is of sufficient similarity to the scenario to 


deserve further examination. The results of such an examination has 


aD. PeeGreeneisen, “A Design Program for Superconducting Elec- 


trical Machines'' (unpublished M. S. thesis, Massachusetts Institute of 


j 
| 
| 
| 
meee ordering which does not include any activities conducted con- 


Technology, 1968), pp. 7-21. 


a6 McMillan and R. F. Gonzalez, Systems Analysis--A Computer 
Approach to Decision Models (Homewood, Illinois: Richard D. Irwin,. 
imc. , 1908), p. 292. 
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revealed that PERT techniques can be used in more complex projects 


where a single scenario may be difficult to en eaiey However, PERT 
analysis will be useful only in those instances where projects can ( 
be sub-divided into appropriately isolated subsidiary networks which 
meet certain criteria.” In the final analysis, thefore, whether one 
considers the scenario to be a special case re general PERT 


technique is relatively immaterial. The only essential characteristics, 


eee 


which must be maintained, are those of sequentiality in the ordering 


ry 


of events, and the selection of events such that technological progress 


iis represented. The importance of the second requirement will become 


obvious as the description proceeds. 


oe ae ee 


After determining the events of the scenario, described above, 


As tem a Oe ee ee ns ee 


' it is necessary to prescribe a procedure for utilizing the data con- 
. tained therein. Towards this end, it is beneficial to use the 
scenario events as milestones for the computation of time and cost 


projections. 


| 
| The development of a time projection may come about in one of 
; two possible ways. First, time until completion may be a given con- 

| straint on the problem. In this case, it is necessary to determine a 
| manning level distribution which will be adequate to meet the time re- 


quirement. The second possibility is that personnel resources may be 





a governing constraint. This, in turn, would require an estimate of 


i 
Rosenbloom, "Notes on the Development of Network Models for 
Resource Allocation R & ) Projects,” p. 62. | 


2p. E.’beckwitn, "A Cost Control Extension of the PERT System," 


IRE Transactions on Engineering Management, EM-9 (December, 1962), p. 
147. 











the time needed for completion with the given personnel available. In 


actuality, a third possibility exists, for which neither personnel 


| 
| 


available nor time is specified. In this instance, a trade-off be- 


tween the two would be sought. The solution for the optimal trade-off 


Regardless of which one of the two situations occurs, the 


: ’ a. ; 1 

is a substantial task in itself, and will not be pursued further, 

determination of manpower or time requirements, from the given con- 
i 


net a a re 


straint, entails an estimate of personnel capability in terms of the 
| Pp p MA 
‘particular project under consideration. Any research organization 


| 
| 
| 


contains a great variety of skills and many levels of efficiency. 
Certain personnel may have had experience which would be particularily 


ae Eomtnceproject Under consideration, If these personnel are 
‘available, completion of the project will be expedited. if, on the 

| other hand, some novice talent is to be used, added time for completion 
Pees be allowed. The primary consideration which is being addressed, 
|here, is that time estimates are functions of the administrator's ex- 
| 


pectations regards the technological demands of the research project, 





and his estimates of the capability of a given research team to cope 





with these demands. 

In addition to the preparation of a time estimate, the scenario 
ae form the foundation of a cost estimate. Ideally, the scenario i 
will be composed of events which represent stages of the project which 


are reasonably different in terms of the activity involved. For 


lror fuiseien consideration of this question see R. A. Goodman, - ~—— 


"Organization and Manpower Utilization in Research and Development," 
IEEE Transactions on Engineering Management, EM-15 (December, 1968), 
Dp. 198, 











instance, in the example scenario which has been presented, the deter- 


mination of operational and theoretical constraints and the development 
of design equations are related to the extent that one is required for 
the accomplishment of the other, and both are dealing with the same 


subject matter. However, the actual activity involved in each, in 


ee ee ee me eee 


terms of solution methods, reference material and theoretical con- 
siderations, will be different. This independence of scenario events 
is not a theoretical imposition, but, simply serves to facilitate 


time and cost analyses. 


Sempre tren ry 


| 
| 
| 
| 
| 


Cost estimates for scenario events are obtained from data con- 
cerning the number and grade of personnel employed, the duration of 


employment, necessary expenditures for equipment and facilities and 


. 
nee 8 ni en es 


appropriate allowances for overhead assignments. This consideration 

of the elements of cost is in keeping with conventional budget prepa- 
ration practices. What is different is the relatively tenuous accuracy 
of the cost estimate. If the administrator, in the preparation of the 


estimate, has improperly assessed any one of several technological 


e 
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‘aspects of the project, the effort anticipated in the funding estimate 
may be entirely inappropriate or inadequate for the task. 

Using the events of the scenario as milestones of progress, 
it is possible to develop cost-achievement and time-achievement 
functions for the project. Before these functions can be expressed, 
a measure of achievement must be determined. Methods of indicating 


achievement are numerous. One such method, in terms of ayi/ay, was 


discussed earlier in this chapter. Due to the diversity of activity 











that is experienced during the course of a project, however, a single 


parameter may not be appropriate. A more useful measure of achievement 
for measuring project progress can be obtained as eoilaege 

Bacneecvent im the scenar1o is assigned a weight, w,, Tepre- 
senting the degree of difficulty associated with the event, measured 


on a 0 to 10 scale. These weights are then multiplied by the number 


iof man-hours assigned to corresponding events, h;. The amount of 
i 


t 
j 
! 
j 


| 
| 
: 
: 





achievement for a given event, a;, is then equal to its weighted 


1? 


number of man-hours divided by the total weighted man-hours. That 9, 


With achievement thus determined, the required functions can then be 


piptted as in Figure 13. 


{The Theoretical Model Model 
Having obtained time-achievement and cost-achievement relation- 
‘ships, the next step is to determine how they may be used in project 
control. If such a decoration is to be made, it will be necessary 
to obtain an analytical model for the research and development process. 
The development of a model can commence with an analysis of the 
time function. To begin, it is recognized that the time for completion 


of any event is a random variable. The estimated value of time, t,;, 


will rarely be the actual time realized. Whether t; is greater or 


] 


Souder, "Experiences with an R & D Project Control Model," 
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smaller than the actual t; will depend on the ability of the person 





making the estimate to anticipate a number of unknown factors in the 


project. These factors include the suitability of the research team 


= ee te ae ate a 


to the task, correct assessment of technological difficulties, and so 
forth. To account for the uncertainty involved, consider a hypo- 


thetical man-day of work. This quantity is a Tayloristic type of 


en i SE ee Ee 


measure and relates to the amount of progress that can be realized 
by a man who is ideally suited for a particular task and devotes 100 
per cent of his time to that task for one working day. Assuming, acon 
that each portion of a project, leading to an event, is composed of 
relatively homogeneous activity, it can then be assumed that each 
event is composed of some number of man-days of Moreover, this 
number of man-days of work is intrinsic to the event itself, given 
EWewState of tne preject from the preceding event. With this concept 


: 2 : a F : . * 
in mind, one realizes that t; 1S an estimate of some t;, the number 


1? 


of calendar days actually required to accomplish the number of man-days 


. 
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of work associated with event i. . 
The quantity t. can be examined further by letting k be the 


number of man-days of work for the event. Then, t; is the number of 





calendar days that will pass before k man-days of work are realized. 
Since projett progress is not constant, the number-of calendar days 
for any given man-day of work is a random variable. Assume, then, 


that there is a probability of p that a man-day of work will be 


accomplished in the calendar interval At. From this, one observes 
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that the variable t; is Poisson distributed. ! ‘ That is, 





p(tysu) = 4 S— (III-21) 


where » is the mean of t;. 

“The Poisson model thus derived can be re Ounce urea : 
world by examining the meaning of the parameters involved. For in- 
stance, k, the number of man-days of work, will be a function of the | 
discipline involved (for example high-energy physics or basic mechan- 
ics) and the degree of achievement to be realized as stated in the 
project objectives. The probability, p, and mean, u, will be functions 
Gestne expertise of the research group and the effectiveness of super- 
lvision.- ine Getrospect, then, 6, is an estimate of a random variable, 
Eyeeeenrors in the estimate; or 1-71, may be due to two factors. One 
factor is the random nature of ti The estimate, at best, 1S an 


ete Of Che Mean, ue. Another factor is a lack of competence or 


| the presence of bias on the part of the person making the estimate. 


| 
| 
| 
| 
| 
i 
| 
i 
| 
| 


Ete eaeteimining a model for the time parameter in the re- 
search and development process, it is not difficult to adopt it to the 
cost factor. Recognizing that direct costs are a strong function of 
project time, one can see that each man-day of work will relate to an 
intrinsic cost. Thus, the cost of a project will also be Poisson 


th 


Richard C. Dubes, The Theory of Applied Probability (Englewood 
Sinees we ew JCrsey..  brentice-Hall, Inc., 1968), pp. 168-70. ; 


siteee 


The use of a Poisson model was suggested by Asher, "A Linear 
Programming Model for the Allocation of R & D Efforts," p. 156. 














distributed and can be expressed by 


p(c;;u) = +—— | (111-22) 
iThe factors affecting the accuracy of cost estimates are the same as 


those that applied to the time parameter. 
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Analysis and Inference 


— 


The ultimate objective of the preceding analysis is the reali- 


me ee ce tee 


oe. 


zation of a method for improving project control techniques. As has 


been discussed at some length, most control techniques are based on 


o_o 


‘prior estimates. These estimates, however, relate to processes that 


— 


are inherently uncertain as to outcome. What is needed, then, is a 


‘method for measuring the accuracy of prior estimates as they relate 


mEQutne actual situation. | 


¢ 
H 


The purpose of developing a scenario is to obtain time and cost 


' 


Comimnares tOnsaeproyect. In doing this, several imtermediate estimates, 


i 
{ 
ror scenario events, are made. In considering these prior estimates, 
the manager is interested in knowing whether the estimates for total 


time, 


T; = ) ty (11-23) 
ot 


and total cost, 


C. =) ¢, (111-24) 
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are accurate within certain bounds. Most control methods delay con- 


clusions, regarding estimate accuracy, until the final stages of a 
project. However, if the mechanism of the time and cost processes are 
iunderstood, such conclusions may be anticipated. 


Mite errors in prvor estimates, that the manager needs to 


identify, are those resulting from incompetence or bias on the part of 


estimates to protect himself from unanticipated problems. Or, if he 
is a perpetual optimist, he may always underestimate cost, or over- 
estimate the ability of the research team. If these errors in the 
total estimate do exist (the fact that is to be determined) it is 
reasonable to expect that proportionate errors will exist in each of 
| 


Givewestimates for the events composing the total. Therefore, as the 
‘first BewecvcCRtsmalcmredchied, the time and costs actually experienced 
“should be useful in anticipating the ultimate outcome. Some techniques 
| which are available for the use of data, obtained subsequent to the 
(prior estimate, can be illustrated by considering the following 
example: Let the data shown in Table 7 be the prior estimates of cost, 
for a project consisting of seven scenario events, and the actual 


costs realized for the first four events. For the moment though, 


assume that only event 1 has been completed. 


After the occurence of event 1, the data in Table 7 is related 


Sf aes real areas en (oe oe 2 al 
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TABLE 7 


PROJECT COST DATA 


eee ee SS ee 


Event Estimate Actual 


ee ae er en at ee A SO a ES rR a, phe gs -ahfSS 
e 
. 
1 
\ 


1 10.0 14.0 

2 14.0 18.5 

3 8.0 i203 | 

4 24.0 32.0 | 

S 31.0 =e | 
| 

6 12.0 = : 

7 9.0 mee | 
| 
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{The objective of a control process, at this point, is to determine 


whether the error 


c= c, = 4.0 | (11-25) 


- ene 


is significant. The pertinent facts which will be used, in determining 


ew 


Significance, are that the random variable, Cy» is Poisson distributed, 


and that c) is the estimate of the mean, y. The appropriate hypothesis — 


in this instance is 
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eae Cp =u <0. 


= 


’ 
ee 


: ASsuming Ho the test to be conducted is to determine the probability 


re ove: 


Menat tie actual results, c*, would occur if 


a 


t 





wee, 
which is! 
© 14.0 
y  _p(ey310) = 2 - YF pley;10) Tibews) 


| 

| ¢)=14.0 ¢4=0 
a: test is accomplished with the use of the cumulative Poisson dis- 
poeaution which is tabulated in Appendix B. Using Equation (III-26), 


’ 


one finds 


J  p(ey310) = 1 - P(14.0;10) 


| 
| 
| 
¢,=14.0 


cles 10 eeS ae 


0.083 
‘ If a confidence limit of 

a= .05 
; had been adopted, then, since 


1ya-tun Chou, Statistical Analysis (New York: Holt, Rinehart 
andevanston, Inc., 1969), p. 209. 


eee ee 
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.083 >a 


the hypothesis, Ho» would be accepted. This amounts to accepting the 


variation of Equation (III-25) as being insignificant due to the ran- 


eee 


dom nature of the variable Cy° 
As more data becomes available with accomplishment of other 
_— as shown in Table 7, a more comprehensive analysis is possible. 


| The type of analysis most commonly employed, with several samples 
2 


a 


test.! This test is especially 


useful in testing Poisson distributed data. Application of the ae test 


| from similar populations, is the x 


j 
|for the data in Table 7 is illustrated in Table 8. | 
| 





: As 1S indicated in Table 8, y? is a measure of the sum of the 


i squares of the variates, as calulated by 


| 


ee | EE SS OE owe 


x2 = ) (cr = Gus ° (I1II-28) 


In order to develop a ee analysis, it is Perec to determine 
the degrees of freedom of the samples. Without pursuing a lengthy 
discussion of the meaning of this quantity, suffice it to say, that, a 
the type of analysis being conducted, when n samples are available, 
there will be n degrees of Eyeeconh 

Io, G. Paradine and B. H. P. Rivett, Statistical Methods for 


Technologists (London: The English Universities Press, Ltd., 1960) 
Dp. 79. 


*Ibid., pp. 76-77. 








99 





et oe ee et Fe ee re 


AO ee a eS eee te ree eee oe = - - mere en 


Bep SATYCIASNITI :992n0S 


> “OnTeA) | {Oo meony eA 
po ewtise peareasao 


VLVG dO SISATVNY 2X 


8 ATaVL 















‘ 


ee 


100 





Phe sresulcs or a ae test are obtained from a comparison of a 
computed value of x7 with a theoretical value determined on the basis 
of the degrees of freedom and the confidence limit, a. Referring to 
the tabulation in Appendix G it is seen that, for four degrees of 
freedom (v=4) and a=.05, then the theoretical value of is 9.49, 


Then, since 


2 Se A UE 
ee 


tS Cc. 
y Mon Eee. < 9.49 (111-29) 
1 


~ 


are Within the limits 


* ~ 


one would conclude that the variations, Cc; - C3, 


of error attributed to the random nature of C,. 


Control Aspects of the Analysis 


Both of the statistical tests, conducted in the preceding 
section, resulted in the conclusion that observed errors between 
estimated and actual expenditures were attributable to random varia- 
tion. However, observing the data in Table 7, one is struck by the 
fact that the test results were obtained from data which demonstrated 
cost overruns ae ComsUepen ccnt.mlt 1s natural, then, to question 
the usefulness of these methods. However, certain factors in the 
analysis should be examined. 


First, it is necessary to recognize the statistical character 


of the Poisson distribution. For instance, the distribution has a 


| 
i] 
| 
t 


pronounced skewness that is a function of the value of the random var- 





iable. In addition to this, the variance is equal to the mean, which 


4 
{ 
| 
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oe 


Beenerally results in substantial dispersion, The end result of these 


‘characteristics is that statistical tests based on Poisson data are 
\ 
? 
,ofter less precise than tests of normally distributed data, for ex- 
4 


jample. From this, one can conclude that conventional confidence limits 


: (for example o=.05 or a=.01), which are generally associated with nor- 


mal data, may be insufficient constraints for tests of Poisson data. 
‘For example, the selection of a=.10 vice .05 would have changed the 
Feeiciusions drawn from the example tests. 

A second important consideration, which must be recognized, re- 
lates to the model which was developed. It must be recalled that the 
data and tests which were conducted were drawn from a model which cor- 


related, for instance, cost and technological achievement. It must 


further be recognized that overruns in research expenditures frequently 


. 
Ae aE re Gh arn ee es = Ce, me Oy So GaN Re ae 


fecemexcced the 50 per cent level. At this point, one might wonder 


fae research and development projects which, after demonstrating ex- 
i cessive costs in early stages, are constrained to complete within final 


ibudget figures. This does frequently occur. However, what invariably 


| 
{ 
Pe nics such forced adherence to budget limitations is a concomi- 
| 
| tant revision of project objectives. Consequently, though expendi- 


| 


‘ture estimates may ultimately be proven to be quite accurate, considcer- 


. |able shortcomings will probably be realized in estimates of achieve- 


| 
{ 
] 
| ment. 
| 
| 
i 


These considerations certainly are significant in any discus- 


‘sion of the usefulness of the statistical techniques that have been 


{ 
{ 
i" 
| developed. It is fairly obvious that two conditions must be satisfied 
‘if such inferential analysis is to be useful. Any organization which 


a 
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might consider such techniques must first of all identify a realistic 
confidence limit, in keeping with what are considered to be acceptable 
amounts of over-run. Second, it is necessary to ensure that achieve- 
ment parameters associated with project events are valid measures of 
technological advance. If this is assured, it must then be accepted 
that attempts to enforce budgetary requirements will likely result in 
a failure to achieve intended project objectives. 

Statistical techniques, such as those examined, do offer add- 
itional methods for controlling research and development. By conduct- 
ing periodic review which incorporates these techniques, additional 


| 
{ 
H 
| 
| 


| information Gane DesObtained Which 1s pertinent to project control. 
‘However, action resulting from such review, whether it be to continue, 
‘terminate, re-evaluate or whatever, must be based on an understanding 
} 


'of what is actually indicated by the results of the analysis. 


! 

| 

{ 

i 

| Summary 

| Chapter II was devoted to the description of numerous methods 
which have been used or have application in project selection and con- 


trol. This chapter has addressed the problem of selecting particular 










techniques, from among those available, which are considered especial- 


technique, not found elsewhere in the literature, has been developed, 
and its applicability in project control has been discussed. Though 
the several ingredients, of the selection and control techniques that 
have been developed, have been drawn from other sources, it is con- 
sidered that the organization of these Mere dienes that has been given 


in this analysis constitutes a new approach to project selection and 


nee oe eens ms eet tees = ee we. 


ieuseruleane tie selection process. in addition, a statistical control 
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control. The procedure for implementing the selection technique has 


been adequately summarized in Figure 11. This technique has emphasized 
the use of information that is generally available in research and 
‘development administrative efforts. It has minimized the use of data 
whach might require special collection efforts. In particular, it has 
javoided the use of estimates of probabilities of success. 

The development of the statistical technique for use in project 
fecuerol has addressed only the basic theoretical concept which forms 
‘its foundation and the demonstration of its use. Actual application of 
te technique has not been pursued because of the variety of methods in 
‘Which it might be used. As has been emphasized, the frequent exclusive 
| 
ence on prior estimates for project control is fraught with un- 
‘certainty. Yet, prior estimates are an essential part of any control 
a The statistical technique developed herein is offered as an 


adjunct to existing control methods. It is considered that this tech- 


ame e  e ve 


ed when combined with usual management centrol practices, will im- 
- the control of research and development programs. This technique 
should enhance management by exception in that it offers more objective 


data for use. This technique is deemed to be particularly desireable 


corporated in the analysis. Presumably, this will serve to minimize 
the need for presumptions regarding progress that are usually drawn 
from an examination of time and cost accounting data. | 

The theoretical treatment that has been offered in this chap- 
ter is noticeably not rigorous. The reason for this is that the com- 


| 
because of the manner in which project achievement is necessarily in- 
vst th ng a re en dee wr re methods of the techniques that have been developed were drawn 
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from other sources and are presumed proven. Emphasis has therefore 
been placed on the organization of these components into a useful, 


coherent system. 


on er a 
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q CHAPTER IV 
| . CONCLUSIONS 

| This paper has attempted to present an analysis of research 

ol development practices with the intent of producing techniques which 
ee be useful in efforts associated with project selection and con- 
ee This effort was prompted by notable Syoreconines in existing 

;filanagement methods as evidenced by widely publicized failures in major 
race. As has been discussed in the preceding chapters, several 


problem areas can be identified as being likely candidates for the 


Meausec or the difficulties that have been experienced. Foremost among 


Ae Ae awe een. 


‘ these is the inadequacy of data employed in present systems. | 

Most contemporary efforts in the management of research and : 
i: development rely on the use of data adopted from established production- 
Pore operations. These data, invariably, emphasize the use of time and 
cost accounting. However, as has been forcibly asserted throughout | 
bthe preceding, this réliance on conventional methods is inadequate; 


due to the nature of research and development activities. One should 


Ae ae RE ee RS 


be able to conclude, by now, that controlling by budget overruns and 
underruns only, can be seriously misleading. It must be emphasized 


that it is the interaction of both cost and achievement that determines 


. 
Fee ene EE te a eS ee EEA A hn RD i US ee rete! ES OEE GN Gemeente an OEE RENOIR <i -* 


/ the actual status of a research project. The conclusion that must be 


| drawn from this is that existing methods of controlling projects on 
} 


. the basis of budgetary variances, which are time and cost functions 
i 
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{ 


alone, are exercises in futility and, at best, will only give the man- 


i 
' 


apex 4 deceptive teeling of satisfaction concerning the conduct of a 


project. This deception is realized only in the latter stages of a 


{ 


‘project Wem 1t isnecessary tO accept the fact that objectives will 


not be accomplished without additional expenditures. It is at this 


t 


point, when most of the available funds have already been spent, that | 


ne wees we em =e 


‘the manager feels compelled to spend more in order to avoid criticism 
i 

for having apparently wasted preceding expenditures. The only method 
available to alleviate such an “eleventh-hour' panic is to continually 


Measure progress throuphout the life of the project. The effectiveness 


‘of such monitoring will, of course, depend on the efficiency of any 


t 


‘analyses of variances which do occur. 


Meanie ctrcmpestGuaccount for the factors of uncertainty which 


i ! 
1 


are frequently encountered, some techniques have relied on a measure 


‘of Bieeprovability Of success of subject projects. It should be 


ee ere 


patently obvious that reliance on such a device pre-empts any efforts 


ito monitor subsequent performance. The probability of success measure 
‘per force assumes that any determination regards success must be re- 


t 


served Until tie conclusion of 2 project. By adopting this technique, 


, the manager neglects his responsibility to assess, in a continuing 


$ 
4 


fasnion, thé successful progress of a project. 


ate at rete. oe 


i 
| These shortcomings would necessarily’ lead one to conclude that 


‘existing methods for the management of research and development are not 
sufficient for their intended purposes. The reasons that more fail- 


, ures than those experienced have not occurred are that management has 


been quite fortunate or research objectives have been sacrificed to 
1 ' 
i 
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avoid overruns. 


The techniques for project selection and control that have 


pbeen presented in this paper, specifically in Chapter III, are regarded 


as offering at least a partial solution for the difficulties that have 
been discussed. The selection technique demands little in terms of the 
jeffort required for implementation. The theory is straight-forward and 
| 


| the assumptions are obvious. The very simplicity of the method will 
i: likely invite criticism that such simplicity cannot possibly account 


H 


.for the numerous intricacies of research and development projects. 


fomoiecriticism Cannot be denied. Indeed, it has been a major premise 


{ 
: , : 2 : 
Of this paper that such intricacies are of paramount importance and 

i 

‘cannot be accounted for in any selection process, regardless of the 

', degree of sophistication that might be offered. However, it must be 


acknowledged that the proposed selection technique is intended for use 


a a 


with the control technique which was the second subject of the pre- 
ceding chapter. 
It has been stressed that any selection method must be based 


on prior estimates. However, efforts to refine such methods of se- 


a renner ee te eres eee eee coe + ane 


lection cannot achieve a degree of certainty beyond the limits imposed 





Jey the inherent nature of the projects being selected. Therefore, 
"| instead of continuing to emphasize the importance of selection, one 
an begin to develop methods to monitor progress so that selection 
ol can be identified in a timely fashion. Selection, Rhea must 


be regarded as only a portion of a system for project management. The 


other very important portion of the system is, of course, control. 


The development and implementation of an effective control 
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technique, relieves, somewhat, the burden placed on selection. If 


effective, control will identify errors that have been made in selection 


'so that funds can be diverted to more successful endeavors. Thus, 


the development of an effective control system is possibly the best 
means of accounting for uncertainty in research and development. 

‘A final aspect of statistical control, that deserves mention, 
lis that of credibility. As was shown in the preceding chapter, it may, 
fat times be difficult to accept the results of analysis. This is, 
however, a difficulty that is fairly common among Baysian-style methods. 
It can be said, in this respect, that such techniques are useful only 
when implemented in an adequate fashion. Attempts to institute such 
“systems on a partial or "trial" basis seldom prove satisfactory since 
| subjective influences invariably offset the prescriptions of objective 
| techniques. 
! These prospects of difficulties in implementation prompt con- 
Piceration of certain aspects of the proposed techniques which deserve 
Seurcher examination and refinement. The first of these is the need to 
| develop a uniformly most powerful test for the Poisson model. The tests 

proposed in this paper are valid for any situation; however, in some 


circumstances -they may be found lacking in precision. It would there- 


fore be advantageous to find a method for structuring data such that 


= see ae ne oe nen 


A second consideration of the model which merits further de- 


1 

| 

| 
the tests will yield consistent results for any circumstances. 
velopment is the need to ensure accurate portrayal of technological 


progress in project scenarios. As has been discussed in preceding 


sections, the measurement of progress in research activities is often 
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quite difficult. However, for the techniques proposed it is absolutely 





‘essential that events be selected such that progress is explicitly in- | 
| | 
dicated. Procedures for identifying adequate indicators and computing 

| | 
| 


‘progress need further development in order that this requrement can be 
Brrisficd. | 
A final aspect which needs more refinement concerns the optimi- 


Zation method that has been selected. To be specific, this method does 


q 


not allow for minimal funding of projects. In many instances, though a 
peeyece may not be included in an optimal funding program, management 

may wish to provide some token appropriation in order that project con- 
‘tinuity is not lost completely. For the proposed technique, though, a 
project is either funded at an adequate level ae Tie ateall. Et there- 


‘fore would be worthwhile to formulate a selection technique which will 


optimize allocations with the provision that a certain number of non- 


e e e e 
‘optimum projects may receive some funding. 


| 
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APPENDIX A 


MISSION-TECHNOLOGY VALUE MATRIX} 


‘An essential part of the selection process is the identifica- 


tion of the contribution of technologies to known objectives. These 


RO 51k Welton 2 oi entree DS Sn ew erence 


matrices were developed for the purpose of relating technologies, 
i 
‘supported by Naval Research efforts, to the research objective areas of 

‘the Naval Materiel Command, Washington, D.C. These matrices are in- 


‘cluded to illustrate the degree of detail required in identifying all 


pertinent relationships. Counterparts to these matrices would be sim- 


‘ilar ones which relate sciences to the technologies given. 
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IMarvin J. Cetron, QUEST Status Report," IEEE Transactions 


on Engineering Management, EM-14 (March, 1967), p. 58. 
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APPENDIX B 


CUMULATIVE POISSON DISTRIBUTION! 


“A tabulation of 
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F(x) = P(c<x) = 


c=0 
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JBernard Ostle, Statistics in Research (Ames, Iowa: The - 
Iowa State University Press, 1963), pp. 372-76. 
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